Jul 30 16:58:23.475736 ip-10-0-143-190 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 30 16:58:23.475747 ip-10-0-143-190 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 30 16:58:23.475754 ip-10-0-143-190 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 30 16:58:23.476189 ip-10-0-143-190 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 30 16:58:33.588881 ip-10-0-143-190 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 30 16:58:33.588899 ip-10-0-143-190 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 0ca1dfe38cb6468b88a491d5106c5fe9 --
Jul 30 17:00:25.397642 ip-10-0-143-190 systemd[1]: Starting Kubernetes Kubelet...
Jul 30 17:00:25.798020 ip-10-0-143-190 kubenswrapper[2555]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 30 17:00:25.798020 ip-10-0-143-190 kubenswrapper[2555]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 30 17:00:25.798020 ip-10-0-143-190 kubenswrapper[2555]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 30 17:00:25.798020 ip-10-0-143-190 kubenswrapper[2555]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 30 17:00:25.798020 ip-10-0-143-190 kubenswrapper[2555]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 30 17:00:25.798725 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.797982    2555 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 30 17:00:25.802254 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802231    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 30 17:00:25.802254 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802254    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 30 17:00:25.802317 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802258    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 30 17:00:25.802317 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802262    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 30 17:00:25.802317 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802265    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 30 17:00:25.802317 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802268    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 30 17:00:25.802317 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802272    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 30 17:00:25.802317 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802275    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 30 17:00:25.802317 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802278    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 30 17:00:25.802317 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802281    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 30 17:00:25.802317 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802284    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 30 17:00:25.802317 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802287    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 30 17:00:25.802317 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802290    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 30 17:00:25.802317 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802292    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 30 17:00:25.802317 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802297    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 30 17:00:25.802317 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802301    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 30 17:00:25.802317 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802304    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 30 17:00:25.802317 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802307    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 30 17:00:25.802317 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802310    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 30 17:00:25.802317 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802313    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 30 17:00:25.802317 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802319    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 30 17:00:25.802782 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802323    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 30 17:00:25.802782 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802326    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 30 17:00:25.802782 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802330    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 30 17:00:25.802782 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802334    2555 feature_gate.go:328] unrecognized feature gate: Example
Jul 30 17:00:25.802782 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802336    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 30 17:00:25.802782 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802339    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 30 17:00:25.802782 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802342    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 30 17:00:25.802782 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802345    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 30 17:00:25.802782 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802348    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 30 17:00:25.802782 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802350    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 30 17:00:25.802782 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802353    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 30 17:00:25.802782 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802355    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 30 17:00:25.802782 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802359    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 30 17:00:25.802782 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802363    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 30 17:00:25.802782 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802366    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 30 17:00:25.802782 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802368    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 30 17:00:25.802782 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802372    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 30 17:00:25.802782 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802374    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 30 17:00:25.802782 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802377    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 30 17:00:25.803278 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802379    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 30 17:00:25.803278 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802382    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 30 17:00:25.803278 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802384    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 30 17:00:25.803278 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802387    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 30 17:00:25.803278 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802389    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 30 17:00:25.803278 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802392    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 30 17:00:25.803278 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802394    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 30 17:00:25.803278 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802397    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 30 17:00:25.803278 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802399    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 30 17:00:25.803278 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802402    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 30 17:00:25.803278 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802404    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 30 17:00:25.803278 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802408    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 30 17:00:25.803278 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802411    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 30 17:00:25.803278 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802413    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 30 17:00:25.803278 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802417    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 30 17:00:25.803278 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802419    2555 feature_gate.go:328] unrecognized feature gate: Example2
Jul 30 17:00:25.803278 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802422    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 30 17:00:25.803278 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802425    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 30 17:00:25.803278 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802427    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 30 17:00:25.803278 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802430    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 30 17:00:25.803773 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802433    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 30 17:00:25.803773 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802435    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 30 17:00:25.803773 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802438    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 30 17:00:25.803773 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802440    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 30 17:00:25.803773 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802443    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 30 17:00:25.803773 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802446    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 30 17:00:25.803773 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802448    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 30 17:00:25.803773 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802451    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 30 17:00:25.803773 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802453    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 30 17:00:25.803773 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802456    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 30 17:00:25.803773 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802458    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 30 17:00:25.803773 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802461    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 30 17:00:25.803773 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802463    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 30 17:00:25.803773 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802466    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 30 17:00:25.803773 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802469    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 30 17:00:25.803773 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802473    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 30 17:00:25.803773 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802476    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 30 17:00:25.803773 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802479    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 30 17:00:25.803773 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802482    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 30 17:00:25.804290 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802484    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 30 17:00:25.804290 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802486    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 30 17:00:25.804290 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802489    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 30 17:00:25.804290 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802491    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 30 17:00:25.804290 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802494    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 30 17:00:25.804290 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802496    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 30 17:00:25.804290 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802501    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 30 17:00:25.804290 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802504    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 30 17:00:25.804290 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802507    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 30 17:00:25.804290 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802510    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 30 17:00:25.804290 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802512    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 30 17:00:25.804290 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802515    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 30 17:00:25.804290 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802518    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 30 17:00:25.804290 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802520    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 30 17:00:25.804290 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802523    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 30 17:00:25.804290 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802525    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 30 17:00:25.804290 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802528    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 30 17:00:25.804290 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802530    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 30 17:00:25.804290 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802533    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 30 17:00:25.804290 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.802536    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 30 17:00:25.804290 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803264    2555 feature_gate.go:328] unrecognized feature gate: Example
Jul 30 17:00:25.804837 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803271    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 30 17:00:25.804837 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803275    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 30 17:00:25.804837 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803278    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 30 17:00:25.804837 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803283    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 30 17:00:25.804837 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803287    2555 feature_gate.go:328] unrecognized feature gate: Example2
Jul 30 17:00:25.804837 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803290    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 30 17:00:25.804837 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803292    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 30 17:00:25.804837 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803295    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 30 17:00:25.804837 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803298    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 30 17:00:25.804837 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803301    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 30 17:00:25.804837 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803304    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 30 17:00:25.804837 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803307    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 30 17:00:25.804837 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803309    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 30 17:00:25.804837 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803312    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 30 17:00:25.804837 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803314    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 30 17:00:25.804837 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803317    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 30 17:00:25.804837 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803320    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 30 17:00:25.804837 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803323    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 30 17:00:25.804837 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803327    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 30 17:00:25.804837 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803330    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 30 17:00:25.805348 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803333    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 30 17:00:25.805348 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803336    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 30 17:00:25.805348 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803339    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 30 17:00:25.805348 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803341    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 30 17:00:25.805348 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803344    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 30 17:00:25.805348 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803346    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 30 17:00:25.805348 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803349    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 30 17:00:25.805348 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803352    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 30 17:00:25.805348 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803354    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 30 17:00:25.805348 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803357    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 30 17:00:25.805348 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803360    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 30 17:00:25.805348 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803362    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 30 17:00:25.805348 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803365    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 30 17:00:25.805348 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803367    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 30 17:00:25.805348 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803371    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 30 17:00:25.805348 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803374    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 30 17:00:25.805348 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803377    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 30 17:00:25.805348 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803380    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 30 17:00:25.805348 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803383    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 30 17:00:25.805348 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803385    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 30 17:00:25.805859 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803387    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 30 17:00:25.805859 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803390    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 30 17:00:25.805859 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803393    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 30 17:00:25.805859 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803395    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 30 17:00:25.805859 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803399    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 30 17:00:25.805859 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803401    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 30 17:00:25.805859 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803404    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 30 17:00:25.805859 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803406    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 30 17:00:25.805859 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803409    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 30 17:00:25.805859 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803413    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 30 17:00:25.805859 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803416    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 30 17:00:25.805859 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803420    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 30 17:00:25.805859 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803422    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 30 17:00:25.805859 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803425    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 30 17:00:25.805859 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803428    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 30 17:00:25.805859 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803430    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 30 17:00:25.805859 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803433    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 30 17:00:25.805859 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803436    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 30 17:00:25.805859 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803438    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 30 17:00:25.805859 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803441    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 30 17:00:25.806369 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803443    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 30 17:00:25.806369 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803446    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 30 17:00:25.806369 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803448    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 30 17:00:25.806369 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803451    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 30 17:00:25.806369 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803453    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 30 17:00:25.806369 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803456    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 30 17:00:25.806369 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803458    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 30 17:00:25.806369 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803461    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 30 17:00:25.806369 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803463    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 30 17:00:25.806369 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803466    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 30 17:00:25.806369 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803469    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 30 17:00:25.806369 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803471    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 30 17:00:25.806369 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803473    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 30 17:00:25.806369 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803476    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 30 17:00:25.806369 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803479    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 30 17:00:25.806369 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803482    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 30 17:00:25.806369 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803484    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 30 17:00:25.806369 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803487    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 30 17:00:25.806369 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803489    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 30 17:00:25.806833 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803492    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 30 17:00:25.806833 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803494    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 30 17:00:25.806833 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803498    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 30 17:00:25.806833 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803501    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 30 17:00:25.806833 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803504    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 30 17:00:25.806833 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803507    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 30 17:00:25.806833 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803509    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 30 17:00:25.806833 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803513    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 30 17:00:25.806833 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803515    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 30 17:00:25.806833 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803518    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 30 17:00:25.806833 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803520    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 30 17:00:25.806833 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803524    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 30 17:00:25.806833 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803527    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 30 17:00:25.806833 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803529    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 30 17:00:25.806833 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803532    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 30 17:00:25.806833 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803535    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 30 17:00:25.806833 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803538    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 30 17:00:25.806833 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803542    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 30 17:00:25.806833 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803545    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 30 17:00:25.807316 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803947    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 30 17:00:25.807316 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803953    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 30 17:00:25.807316 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803956    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 30 17:00:25.807316 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803959    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 30 17:00:25.807316 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803963    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 30 17:00:25.807316 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803967    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 30 17:00:25.807316 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803970    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 30 17:00:25.807316 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803973    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 30 17:00:25.807316 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803976    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 30 17:00:25.807316 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803978    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 30 17:00:25.807316 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803981    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 30 17:00:25.807316 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803983    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 30 17:00:25.807316 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803986    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 30 17:00:25.807316 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803989    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 30 17:00:25.807316 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803992    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 30 17:00:25.807316 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803995    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 30 17:00:25.807316 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.803998    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 30 17:00:25.807316 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804001    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 30 17:00:25.807316 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804017    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 30 17:00:25.807775 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804020    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 30 17:00:25.807775 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804023    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 30 17:00:25.807775 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804026    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 30 17:00:25.807775 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804029    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 30 17:00:25.807775 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804031    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 30 17:00:25.807775 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804034    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 30 17:00:25.807775 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804038    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 30 17:00:25.807775 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804041    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 30 17:00:25.807775 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804044    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 30 17:00:25.807775 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804047    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 30 17:00:25.807775 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804050    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 30 17:00:25.807775 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804052    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 30 17:00:25.807775 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804055    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 30 17:00:25.807775 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804057    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 30 17:00:25.807775 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804060    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 30 17:00:25.807775 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804062    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 30 17:00:25.807775 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804065    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 30 17:00:25.807775 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804067    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 30 17:00:25.807775 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804070    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 30 17:00:25.807775 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804073    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 30 17:00:25.808293 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804075    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 30 17:00:25.808293 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804078    2555 feature_gate.go:328] unrecognized feature gate: Example
Jul 30 17:00:25.808293 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804080    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 30 17:00:25.808293 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804083    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 30 17:00:25.808293 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804085    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 30 17:00:25.808293 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804088    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 30 17:00:25.808293 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804090    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 30 17:00:25.808293 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804093    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 30 17:00:25.808293 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804096    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 30 17:00:25.808293 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804098    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 30 17:00:25.808293 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804101    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 30 17:00:25.808293 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804103    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 30 17:00:25.808293 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804106    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 30 17:00:25.808293 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804109    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 30 17:00:25.808293 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804112    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 30 17:00:25.808293 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804114    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 30 17:00:25.808293 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804117    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 30 17:00:25.808293 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804119    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 30 17:00:25.808293 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804122    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 30 17:00:25.808293 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804124    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 30 17:00:25.808811 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804127    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 30 17:00:25.808811 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804129    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 30 17:00:25.808811 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804132    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 30 17:00:25.808811 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804134    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 30 17:00:25.808811 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804136    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 30 17:00:25.808811 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804139    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 30 17:00:25.808811 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804141    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 30 17:00:25.808811 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804144    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 30 17:00:25.808811 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804146    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 30 17:00:25.808811 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804149    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 30 17:00:25.808811 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804152    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 30 17:00:25.808811 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804155    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 30 17:00:25.808811 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804158    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 30 17:00:25.808811 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804160    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 30 17:00:25.808811 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804163    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 30 17:00:25.808811 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804165    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 30 17:00:25.808811 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804168    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 30 17:00:25.808811 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804170    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 30 17:00:25.808811 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804173    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 30 17:00:25.808811 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804175    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 30 17:00:25.809364 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804180    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 30 17:00:25.809364 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804182    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 30 17:00:25.809364 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804185    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 30 17:00:25.809364 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804188    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 30 17:00:25.809364 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804190    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 30 17:00:25.809364 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804193    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 30 17:00:25.809364 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804196    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 30 17:00:25.809364 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804199    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 30 17:00:25.809364 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804201    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 30 17:00:25.809364 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804204    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 30 17:00:25.809364 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804206    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 30 17:00:25.809364 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804209    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 30 17:00:25.809364 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804213    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 30 17:00:25.809364 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804217    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 30 17:00:25.809364 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804220    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 30 17:00:25.809364 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804223    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 30 17:00:25.809364 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804226    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 30 17:00:25.809364 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804228    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 30 17:00:25.809364 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804231    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 30 17:00:25.809925 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804233    2555 feature_gate.go:328] unrecognized feature gate: Example2
Jul 30 17:00:25.809925 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804312    2555 flags.go:64] FLAG: --address="0.0.0.0"
Jul 30 17:00:25.809925 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804325    2555 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 30 17:00:25.809925 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804332    2555 flags.go:64] FLAG: --anonymous-auth="true"
Jul 30 17:00:25.809925 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804339    2555 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 30 17:00:25.809925 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804345    2555 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 30 17:00:25.809925 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804350    2555 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 30 17:00:25.809925 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804356    2555 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 30 17:00:25.809925 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804363    2555 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 30 17:00:25.809925 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804368    2555 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 30 17:00:25.809925 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804371    2555 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 30 17:00:25.809925 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804375    2555 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 30 17:00:25.809925 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804378    2555 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 30 17:00:25.809925 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804383    2555 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 30 17:00:25.809925 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804386    2555 flags.go:64] FLAG: --cgroup-root=""
Jul 30 17:00:25.809925 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804389    2555 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 30 17:00:25.809925 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804392    2555 flags.go:64] FLAG: --client-ca-file=""
Jul 30 17:00:25.809925 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804395    2555 flags.go:64] FLAG: --cloud-provider="external"
Jul 30 17:00:25.809925 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804398    2555 flags.go:64] FLAG: --cluster-dns="[]"
Jul 30 17:00:25.809925 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804405    2555 flags.go:64] FLAG: --cluster-domain=""
Jul 30 17:00:25.809925 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804408    2555 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 30 17:00:25.809925 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804411    2555 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 30 17:00:25.809925 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804414    2555 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 30 17:00:25.809925 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804417    2555 flags.go:64] FLAG: --container-log-max-files="5"
Jul 30 17:00:25.810523 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804421    2555 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 30 17:00:25.810523 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804424    2555 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 30 17:00:25.810523 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804428    2555 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 30 17:00:25.810523 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804431    2555 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 30 17:00:25.810523 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804434    2555 flags.go:64] FLAG: --contention-profiling="false"
Jul 30 17:00:25.810523 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804438    2555 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 30 17:00:25.810523 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804440    2555 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 30 17:00:25.810523 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804443    2555 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 30 17:00:25.810523 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804446    2555 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 30 17:00:25.810523 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804451    2555 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 30 17:00:25.810523 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804454    2555 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 30 17:00:25.810523 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804457    2555 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 30 17:00:25.810523 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804461    2555 flags.go:64] FLAG: --enable-load-reader="false"
Jul 30 17:00:25.810523 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804464    2555 flags.go:64] FLAG: --enable-server="true"
Jul 30 17:00:25.810523 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804467    2555 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 30 17:00:25.810523 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804472    2555 flags.go:64] FLAG: --event-burst="100"
Jul 30 17:00:25.810523 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804475    2555 flags.go:64] FLAG: --event-qps="50"
Jul 30 17:00:25.810523 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804478    2555 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 30 17:00:25.810523 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804481    2555 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 30 17:00:25.810523 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804484    2555 flags.go:64] FLAG: --eviction-hard=""
Jul 30 17:00:25.810523 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804488    2555 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 30 17:00:25.810523 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804493    2555 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 30 17:00:25.810523 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804496    2555 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 30 17:00:25.810523 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804499    2555 flags.go:64] FLAG: --eviction-soft=""
Jul 30 17:00:25.810523 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804510    2555 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 30 17:00:25.811144 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804514    2555 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 30 17:00:25.811144 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804518    2555 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 30 17:00:25.811144 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804521    2555 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 30 17:00:25.811144 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804524    2555 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 30 17:00:25.811144 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804527    2555 flags.go:64] FLAG: --fail-swap-on="true"
Jul 30 17:00:25.811144 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804531    2555 flags.go:64] FLAG: --feature-gates=""
Jul 30 17:00:25.811144 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804535    2555 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 30 17:00:25.811144 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804538    2555 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 30 17:00:25.811144 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804541    2555 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 30 17:00:25.811144 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804544    2555 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 30 17:00:25.811144 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804547    2555 flags.go:64] FLAG: --healthz-port="10248"
Jul 30 17:00:25.811144 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804550    2555 flags.go:64] FLAG: --help="false"
Jul 30 17:00:25.811144 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804553    2555 flags.go:64] FLAG: --hostname-override="ip-10-0-143-190.ec2.internal"
Jul 30 17:00:25.811144 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804556    2555 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 30 17:00:25.811144 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804559    2555 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 30 17:00:25.811144 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804562    2555 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 30 17:00:25.811144 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804566    2555 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 30 17:00:25.811144 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804569    2555 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 30 17:00:25.811144 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804572    2555 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 30 17:00:25.811144 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804576    2555 flags.go:64] FLAG: --image-service-endpoint=""
Jul 30 17:00:25.811144 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804579    2555 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 30 17:00:25.811144 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804582    2555 flags.go:64] FLAG: --kube-api-burst="100"
Jul 30 17:00:25.811144 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804585    2555 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 30 17:00:25.811807 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804588    2555 flags.go:64] FLAG: --kube-api-qps="50"
Jul 30 17:00:25.811807 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804591    2555 flags.go:64] FLAG: --kube-reserved=""
Jul 30 17:00:25.811807 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804594    2555 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 30 17:00:25.811807 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804597    2555 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 30 17:00:25.811807 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804600    2555 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 30 17:00:25.811807 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804604    2555 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 30 17:00:25.811807 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804607    2555 flags.go:64] FLAG: --lock-file=""
Jul 30 17:00:25.811807 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804610    2555 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 30 17:00:25.811807 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804613    2555 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 30 17:00:25.811807 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804616    2555 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 30 17:00:25.811807 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804622    2555 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 30 17:00:25.811807 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804625    2555 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 30 17:00:25.811807 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804628    2555 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 30 17:00:25.811807 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804631    2555 flags.go:64] FLAG: --logging-format="text"
Jul 30 17:00:25.811807 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804635    2555 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 30 17:00:25.811807 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804638    2555 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 30 17:00:25.811807 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804641    2555 flags.go:64] FLAG: --manifest-url=""
Jul 30 17:00:25.811807 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804644    2555 flags.go:64] FLAG: --manifest-url-header=""
Jul 30 17:00:25.811807 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804648    2555 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 30 17:00:25.811807 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804651    2555 flags.go:64] FLAG: --max-open-files="1000000"
Jul 30 17:00:25.811807 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804655    2555 flags.go:64] FLAG: --max-pods="110"
Jul 30 17:00:25.811807 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804658    2555 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 30 17:00:25.811807 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804661    2555 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 30 17:00:25.811807 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804664    2555 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 30 17:00:25.811807 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804667    2555 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 30 17:00:25.812604 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804670    2555 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 30 17:00:25.812604 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804673    2555 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 30 17:00:25.812604 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804676    2555 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 30 17:00:25.812604 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804685    2555 flags.go:64] FLAG: --node-status-max-images="50"
Jul 30 17:00:25.812604 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804689    2555 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 30 17:00:25.812604 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804692    2555 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 30 17:00:25.812604 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804695    2555 flags.go:64] FLAG: --pod-cidr=""
Jul 30 17:00:25.812604 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804698    2555 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 30 17:00:25.812604 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804704    2555 flags.go:64] FLAG: --pod-manifest-path=""
Jul 30 17:00:25.812604 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804707    2555 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 30 17:00:25.812604 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804710    2555 flags.go:64] FLAG: --pods-per-core="0"
Jul 30 17:00:25.812604 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804713    2555 flags.go:64] FLAG: --port="10250"
Jul 30 17:00:25.812604 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804716    2555 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 30 17:00:25.812604 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804721    2555 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d0d6a41a9098f063"
Jul 30 17:00:25.812604 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804724    2555 flags.go:64] FLAG: --qos-reserved=""
Jul 30 17:00:25.812604 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804727    2555 flags.go:64] FLAG: --read-only-port="10255"
Jul 30 17:00:25.812604 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804730    2555 flags.go:64] FLAG: --register-node="true"
Jul 30 17:00:25.812604 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804733    2555 flags.go:64] FLAG: --register-with-taints=""
Jul 30 17:00:25.812604 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804737    2555 flags.go:64] FLAG: --registry-burst="10"
Jul 30 17:00:25.812604 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804740    2555 flags.go:64] FLAG: --registry-qps="5"
Jul 30 17:00:25.812604 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804743    2555 flags.go:64] FLAG: --reserved-cpus=""
Jul 30 17:00:25.812604 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804747    2555 flags.go:64] FLAG: --reserved-memory=""
Jul 30 17:00:25.812604 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804751    2555 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 30 17:00:25.812604 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804754    2555 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 30 17:00:25.812604 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804757    2555 flags.go:64] FLAG: --rotate-certificates="false"
Jul 30 17:00:25.813577 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804760    2555 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 30 17:00:25.813577 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804763    2555 flags.go:64] FLAG: --runonce="false"
Jul 30 17:00:25.813577 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804766    2555 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 30 17:00:25.813577 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804769    2555 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 30 17:00:25.813577 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804772    2555 flags.go:64] FLAG: --seccomp-default="false"
Jul 30 17:00:25.813577 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804774    2555 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 30 17:00:25.813577 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804777    2555 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 30 17:00:25.813577 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804781    2555 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 30 17:00:25.813577 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804784    2555 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 30 17:00:25.813577 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804787    2555 flags.go:64] FLAG: --storage-driver-password="root"
Jul 30 17:00:25.813577 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804792    2555 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 30 17:00:25.813577 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804795    2555 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 30 17:00:25.813577 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804798    2555 flags.go:64] FLAG: --storage-driver-user="root"
Jul 30 17:00:25.813577 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804801    2555 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 30 17:00:25.813577 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804804    2555 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 30 17:00:25.813577 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804807    2555 flags.go:64] FLAG: --system-cgroups=""
Jul 30 17:00:25.813577 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804809    2555 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 30 17:00:25.813577 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804815    2555 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 30 17:00:25.813577 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804817    2555 flags.go:64] FLAG: --tls-cert-file=""
Jul 30 17:00:25.813577 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804820    2555 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 30 17:00:25.813577 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804827    2555 flags.go:64] FLAG: --tls-min-version=""
Jul 30 17:00:25.813577 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804830    2555 flags.go:64] FLAG: --tls-private-key-file=""
Jul 30 17:00:25.813577 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804832    2555 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 30 17:00:25.813577 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804835    2555 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 30 17:00:25.813577 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804838    2555 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 30 17:00:25.814342 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804842    2555 flags.go:64] FLAG: --v="2"
Jul 30 17:00:25.814342 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804847    2555 flags.go:64] FLAG: --version="false"
Jul 30 17:00:25.814342 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804851    2555 flags.go:64] FLAG: --vmodule=""
Jul 30 17:00:25.814342 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804855    2555 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 30 17:00:25.814342 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.804858    2555 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 30 17:00:25.814342 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804980    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 30 17:00:25.814342 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804984    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 30 17:00:25.814342 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804988    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 30 17:00:25.814342 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804991    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 30 17:00:25.814342 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804993    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 30 17:00:25.814342 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.804998    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 30 17:00:25.814342 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805002    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 30 17:00:25.814342 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805016    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 30 17:00:25.814342 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805019    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 30 17:00:25.814342 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805022    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 30 17:00:25.814342 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805024    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 30 17:00:25.814342 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805027    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 30 17:00:25.814342 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805030    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 30 17:00:25.814342 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805033    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 30 17:00:25.814342 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805036    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 30 17:00:25.814342 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805038    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 30 17:00:25.814860 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805041    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 30 17:00:25.814860 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805044    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 30 17:00:25.814860 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805047    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 30 17:00:25.814860 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805050    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 30 17:00:25.814860 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805052    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 30 17:00:25.814860 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805055    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 30 17:00:25.814860 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805060    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 30 17:00:25.814860 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805062    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 30 17:00:25.814860 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805065    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 30 17:00:25.814860 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805068    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 30 17:00:25.814860 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805070    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 30 17:00:25.814860 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805073    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 30 17:00:25.814860 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805076    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 30 17:00:25.814860 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805078    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 30 17:00:25.814860 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805082    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 30 17:00:25.814860 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805084    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 30 17:00:25.814860 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805088    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 30 17:00:25.814860 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805091    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 30 17:00:25.814860 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805093    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 30 17:00:25.814860 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805096    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 30 17:00:25.815374 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805099    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 30 17:00:25.815374 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805101    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 30 17:00:25.815374 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805104    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 30 17:00:25.815374 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805106    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 30 17:00:25.815374 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805109    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 30 17:00:25.815374 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805111    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 30 17:00:25.815374 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805114    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 30 17:00:25.815374 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805117    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 30 17:00:25.815374 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805120    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 30 17:00:25.815374 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805122    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 30 17:00:25.815374 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805125    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 30 17:00:25.815374 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805127    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 30 17:00:25.815374 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805130    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 30 17:00:25.815374 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805132    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 30 17:00:25.815374 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805135    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 30 17:00:25.815374 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805139    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 30 17:00:25.815374 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805142    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 30 17:00:25.815374 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805145    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 30 17:00:25.815374 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805150    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 30 17:00:25.815849 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805153    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 30 17:00:25.815849 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805156    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 30 17:00:25.815849 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805158    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 30 17:00:25.815849 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805161    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 30 17:00:25.815849 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805163    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 30 17:00:25.815849 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805166    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 30 17:00:25.815849 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805169    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 30 17:00:25.815849 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805171    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 30 17:00:25.815849 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805174    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 30 17:00:25.815849 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805176    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 30 17:00:25.815849 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805179    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 30 17:00:25.815849 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805182    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 30 17:00:25.815849 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805185    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 30 17:00:25.815849 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805187    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 30 17:00:25.815849 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805190    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 30 17:00:25.815849 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805193    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 30 17:00:25.815849 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805195    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 30 17:00:25.815849 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805198    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 30 17:00:25.815849 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805202    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 30 17:00:25.816322 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805205    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 30 17:00:25.816322 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805208    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 30 17:00:25.816322 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805211    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 30 17:00:25.816322 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805214    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 30 17:00:25.816322 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805217    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 30 17:00:25.816322 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805220    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 30 17:00:25.816322 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805223    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 30 17:00:25.816322 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805225    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 30 17:00:25.816322 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805228    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 30 17:00:25.816322 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805230    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 30 17:00:25.816322 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805233    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 30 17:00:25.816322 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805235    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 30 17:00:25.816322 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805240    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 30 17:00:25.816322 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805243    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 30 17:00:25.816322 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805246    2555 feature_gate.go:328] unrecognized feature gate: Example2
Jul 30 17:00:25.816322 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805248    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 30 17:00:25.816322 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805251    2555 feature_gate.go:328] unrecognized feature gate: Example
Jul 30 17:00:25.816322 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805254    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 30 17:00:25.816322 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805256    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 30 17:00:25.816322 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805259    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 30 17:00:25.816821 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805261    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 30 17:00:25.816821 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805264    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 30 17:00:25.816821 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805266    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 30 17:00:25.816821 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805269    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 30 17:00:25.816821 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.805272    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 30 17:00:25.816821 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.805851    2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 30 17:00:25.816821 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.814255    2555 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 30 17:00:25.816821 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.814276    2555 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 30 17:00:25.816821 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.814298    2555 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 30 17:00:25.816821 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.814307    2555 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 30 17:00:25.816821 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814369    2555 feature_gate.go:328] unrecognized feature gate: Example
Jul 30 17:00:25.816821 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814374    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 30 17:00:25.816821 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814378    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 30 17:00:25.816821 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814381    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 30 17:00:25.816821 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814383    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 30 17:00:25.817219 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814386    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 30 17:00:25.817219 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814390    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 30 17:00:25.817219 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814393    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 30 17:00:25.817219 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814395    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 30 17:00:25.817219 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814398    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 30 17:00:25.817219 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814402    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 30 17:00:25.817219 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814404    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 30 17:00:25.817219 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814409    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 30 17:00:25.817219 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814413    2555 feature_gate.go:328] unrecognized feature gate: Example2
Jul 30 17:00:25.817219 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814416    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 30 17:00:25.817219 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814420    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 30 17:00:25.817219 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814423    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 30 17:00:25.817219 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814426    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 30 17:00:25.817219 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814428    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 30 17:00:25.817219 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814432    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 30 17:00:25.817219 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814434    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 30 17:00:25.817219 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814437    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 30 17:00:25.817219 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814440    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 30 17:00:25.817219 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814442    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 30 17:00:25.817684 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814445    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 30 17:00:25.817684 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814448    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 30 17:00:25.817684 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814450    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 30 17:00:25.817684 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814453    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 30 17:00:25.817684 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814456    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 30 17:00:25.817684 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814460    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 30 17:00:25.817684 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814463    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 30 17:00:25.817684 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814466    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 30 17:00:25.817684 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814468    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 30 17:00:25.817684 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814471    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 30 17:00:25.817684 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814474    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 30 17:00:25.817684 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814477    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 30 17:00:25.817684 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814480    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 30 17:00:25.817684 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814482    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 30 17:00:25.817684 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814485    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 30 17:00:25.817684 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814487    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 30 17:00:25.817684 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814490    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 30 17:00:25.817684 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814493    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 30 17:00:25.817684 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814495    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 30 17:00:25.817684 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814498    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 30 17:00:25.818178 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814500    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 30 17:00:25.818178 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814502    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 30 17:00:25.818178 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814505    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 30 17:00:25.818178 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814508    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 30 17:00:25.818178 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814511    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 30 17:00:25.818178 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814513    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 30 17:00:25.818178 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814516    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 30 17:00:25.818178 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814519    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 30 17:00:25.818178 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814521    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 30 17:00:25.818178 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814524    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 30 17:00:25.818178 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814526    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 30 17:00:25.818178 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814529    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 30 17:00:25.818178 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814531    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 30 17:00:25.818178 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814534    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 30 17:00:25.818178 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814536    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 30 17:00:25.818178 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814539    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 30 17:00:25.818178 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814542    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 30 17:00:25.818178 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814545    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 30 17:00:25.818178 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814548    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 30 17:00:25.818178 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814550    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 30 17:00:25.818663 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814553    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 30 17:00:25.818663 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814555    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 30 17:00:25.818663 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814558    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 30 17:00:25.818663 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814560    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 30 17:00:25.818663 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814563    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 30 17:00:25.818663 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814567    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 30 17:00:25.818663 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814571    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 30 17:00:25.818663 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814574    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 30 17:00:25.818663 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814577    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 30 17:00:25.818663 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814580    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 30 17:00:25.818663 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814583    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 30 17:00:25.818663 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814586    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 30 17:00:25.818663 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814589    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 30 17:00:25.818663 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814592    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 30 17:00:25.818663 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814595    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 30 17:00:25.818663 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814599    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 30 17:00:25.818663 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814603    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 30 17:00:25.818663 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814606    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 30 17:00:25.818663 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814609    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 30 17:00:25.819185 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814612    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 30 17:00:25.819185 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814615    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 30 17:00:25.819185 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814618    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 30 17:00:25.819185 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814621    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 30 17:00:25.819185 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814623    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 30 17:00:25.819185 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814626    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 30 17:00:25.819185 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814629    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 30 17:00:25.819185 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814631    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 30 17:00:25.819185 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814634    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 30 17:00:25.819185 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814636    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 30 17:00:25.819185 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814639    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 30 17:00:25.819185 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814642    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 30 17:00:25.819185 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814644    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 30 17:00:25.819185 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814647    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 30 17:00:25.819185 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814649    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 30 17:00:25.819185 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814652    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 30 17:00:25.819581 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.814658    2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 30 17:00:25.819581 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814775    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 30 17:00:25.819581 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814780    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 30 17:00:25.819581 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814784    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 30 17:00:25.819581 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814787    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 30 17:00:25.819581 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814790    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 30 17:00:25.819581 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814793    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 30 17:00:25.819581 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814796    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 30 17:00:25.819581 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814799    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 30 17:00:25.819581 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814801    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 30 17:00:25.819581 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814804    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 30 17:00:25.819581 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814807    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 30 17:00:25.819581 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814810    2555 feature_gate.go:328] unrecognized feature gate: Example
Jul 30 17:00:25.819581 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814813    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 30 17:00:25.819581 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814816    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 30 17:00:25.819944 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814819    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 30 17:00:25.819944 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814821    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 30 17:00:25.819944 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814824    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 30 17:00:25.819944 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814827    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 30 17:00:25.819944 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814829    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 30 17:00:25.819944 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814832    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 30 17:00:25.819944 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814835    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 30 17:00:25.819944 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814837    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 30 17:00:25.819944 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814841    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 30 17:00:25.819944 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814845    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 30 17:00:25.819944 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814848    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 30 17:00:25.819944 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814851    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 30 17:00:25.819944 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814854    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 30 17:00:25.819944 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814856    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 30 17:00:25.819944 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814859    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 30 17:00:25.819944 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814862    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 30 17:00:25.819944 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814865    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 30 17:00:25.819944 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814868    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 30 17:00:25.819944 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814871    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 30 17:00:25.819944 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814874    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 30 17:00:25.820440 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814878    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 30 17:00:25.820440 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814881    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 30 17:00:25.820440 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814884    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 30 17:00:25.820440 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814887    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 30 17:00:25.820440 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814890    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 30 17:00:25.820440 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814892    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 30 17:00:25.820440 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814895    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 30 17:00:25.820440 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814898    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 30 17:00:25.820440 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814900    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 30 17:00:25.820440 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814903    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 30 17:00:25.820440 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814905    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 30 17:00:25.820440 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814908    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 30 17:00:25.820440 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814911    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 30 17:00:25.820440 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814913    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 30 17:00:25.820440 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814916    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 30 17:00:25.820440 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814919    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 30 17:00:25.820440 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814922    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 30 17:00:25.820440 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814924    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 30 17:00:25.820440 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814927    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 30 17:00:25.820910 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814930    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 30 17:00:25.820910 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814932    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 30 17:00:25.820910 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814935    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 30 17:00:25.820910 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814937    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 30 17:00:25.820910 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814942    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 30 17:00:25.820910 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814945    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 30 17:00:25.820910 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814949    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 30 17:00:25.820910 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814951    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 30 17:00:25.820910 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814955    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 30 17:00:25.820910 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814957    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 30 17:00:25.820910 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814960    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 30 17:00:25.820910 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814963    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 30 17:00:25.820910 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814966    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 30 17:00:25.820910 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814969    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 30 17:00:25.820910 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814971    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 30 17:00:25.820910 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814974    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 30 17:00:25.820910 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814977    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 30 17:00:25.820910 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814979    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 30 17:00:25.820910 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814982    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 30 17:00:25.820910 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814985    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 30 17:00:25.821434 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814987    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 30 17:00:25.821434 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814990    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 30 17:00:25.821434 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814992    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 30 17:00:25.821434 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814995    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 30 17:00:25.821434 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.814998    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 30 17:00:25.821434 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.815001    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 30 17:00:25.821434 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.815021    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 30 17:00:25.821434 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.815026    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 30 17:00:25.821434 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.815029    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 30 17:00:25.821434 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.815031    2555 feature_gate.go:328] unrecognized feature gate: Example2
Jul 30 17:00:25.821434 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.815034    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 30 17:00:25.821434 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.815037    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 30 17:00:25.821434 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.815039    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 30 17:00:25.821434 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.815042    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 30 17:00:25.821434 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.815045    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 30 17:00:25.821434 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.815047    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 30 17:00:25.821434 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.815050    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 30 17:00:25.821434 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.815053    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 30 17:00:25.821434 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.815056    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 30 17:00:25.821434 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.815059    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 30 17:00:25.821902 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.815062    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 30 17:00:25.821902 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.815065    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 30 17:00:25.821902 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.815067    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 30 17:00:25.821902 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.815070    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 30 17:00:25.821902 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.815073    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 30 17:00:25.821902 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:25.815075    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 30 17:00:25.821902 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.815081    2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 30 17:00:25.821902 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.815214    2555 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 30 17:00:25.821902 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.819326    2555 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 30 17:00:25.821902 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.820538    2555 server.go:1019] "Starting client certificate rotation"
Jul 30 17:00:25.821902 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.820637    2555 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 30 17:00:25.821902 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.820672    2555 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 30 17:00:25.822812 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.822797    2555 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 30 17:00:25.824894 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.824880    2555 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 30 17:00:25.835974 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.835952    2555 log.go:25] "Validated CRI v1 runtime API"
Jul 30 17:00:25.838433 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.838415    2555 log.go:25] "Validated CRI v1 image API"
Jul 30 17:00:25.839875 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.839860    2555 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 30 17:00:25.842229 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.842201    2555 fs.go:135] Filesystem UUIDs: map[180bb26b-9c64-41a7-93a4-3969aa314575:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 9ffb488e-238c-45a9-8304-e715534d4324:/dev/nvme0n1p4]
Jul 30 17:00:25.842319 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.842226    2555 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 30 17:00:25.847647 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.847625    2555 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 30 17:00:25.848853 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.848746    2555 manager.go:216] Machine: {Timestamp:2026-07-30 17:00:25.847374347 +0000 UTC m=+0.334741739 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099449 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec213881a041a7bdbef9f51079b77e92 SystemUUID:ec213881-a041-a7bd-bef9-f51079b77e92 BootID:0ca1dfe3-8cb6-468b-88a4-91d5106c5fe9 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:0a:4b:70:c1:eb Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:0a:4b:70:c1:eb Speed:0 Mtu:9001} {Name:ovs-system MacAddress:22:7e:fd:54:94:2e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 30 17:00:25.848853 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.848846    2555 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 30 17:00:25.848994 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.848958    2555 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 30 17:00:25.852542 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.852350    2555 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 30 17:00:25.852686 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.852544    2555 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-190.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 30 17:00:25.853224 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.853215    2555 topology_manager.go:138] "Creating topology manager with none policy"
Jul 30 17:00:25.853258 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.853226    2555 container_manager_linux.go:309] "Creating device plugin manager"
Jul 30 17:00:25.853258 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.853233    2555 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 30 17:00:25.854086 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.854075    2555 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 30 17:00:25.854902 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.854893    2555 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 30 17:00:25.856571 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.856560    2555 state_mem.go:36] "Initialized new in-memory state store"
Jul 30 17:00:25.856695 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.856686    2555 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 30 17:00:25.856724 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.856718    2555 kubelet.go:486] "Attempting to sync node with API server"
Jul 30 17:00:25.856755 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.856727    2555 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 30 17:00:25.856755 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.856738    2555 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 30 17:00:25.856755 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.856747    2555 kubelet.go:398] "Adding apiserver pod source"
Jul 30 17:00:25.856755 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.856755    2555 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 30 17:00:25.857161 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.857150    2555 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 30 17:00:25.857214 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.857170    2555 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 30 17:00:25.859481 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.859461    2555 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 30 17:00:25.861226 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.861212    2555 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 30 17:00:25.861270 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.861242    2555 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 30 17:00:25.862675 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.862664    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 30 17:00:25.862719 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.862681    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 30 17:00:25.862719 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.862687    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 30 17:00:25.862719 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.862694    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 30 17:00:25.862719 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.862708    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 30 17:00:25.862719 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.862715    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 30 17:00:25.862860 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.862723    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 30 17:00:25.862860 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.862728    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 30 17:00:25.862860 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.862735    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 30 17:00:25.862860 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.862740    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 30 17:00:25.862860 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.862749    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 30 17:00:25.862860 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.862754    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 30 17:00:25.863560 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.863550    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 30 17:00:25.863560 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.863559    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 30 17:00:25.867135 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.867122    2555 server.go:1268] "Started kubelet"
Jul 30 17:00:25.867272 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.867221    2555 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 30 17:00:25.867470 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.867403    2555 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 30 17:00:25.867546 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.867495    2555 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 30 17:00:25.867781 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.867760    2555 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-190.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 30 17:00:25.867781 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:25.867761    2555 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 30 17:00:25.867891 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:25.867813    2555 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-190.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 30 17:00:25.868035 ip-10-0-143-190 systemd[1]: Started Kubernetes Kubelet.
Jul 30 17:00:25.868968 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.868952    2555 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 30 17:00:25.869217 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.869176    2555 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hd6wm"
Jul 30 17:00:25.870085 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.870068    2555 server.go:310] "Adding debug handlers to kubelet server"
Jul 30 17:00:25.872771 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:25.872741    2555 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 30 17:00:25.875178 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.875163    2555 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 30 17:00:25.875248 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.875187    2555 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 30 17:00:25.875737 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:25.875704    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-190.ec2.internal\" not found"
Jul 30 17:00:25.875922 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:25.874391    2555 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-190.ec2.internal.18c71fe062d726c7  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-190.ec2.internal,UID:ip-10-0-143-190.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-190.ec2.internal,},FirstTimestamp:2026-07-30 17:00:25.867093703 +0000 UTC m=+0.354461095,LastTimestamp:2026-07-30 17:00:25.867093703 +0000 UTC m=+0.354461095,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-190.ec2.internal,}"
Jul 30 17:00:25.876028 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.875951    2555 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 30 17:00:25.876028 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.875965    2555 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 30 17:00:25.876139 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.876030    2555 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 30 17:00:25.876139 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.876030    2555 factory.go:55] Registering systemd factory
Jul 30 17:00:25.876139 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.876053    2555 factory.go:223] Registration of the systemd container factory successfully
Jul 30 17:00:25.876139 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.876104    2555 reconstruct.go:97] "Volume reconstruction finished"
Jul 30 17:00:25.876139 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.876114    2555 reconciler.go:29] "Reconciler: start to sync state"
Jul 30 17:00:25.876344 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.876257    2555 factory.go:153] Registering CRI-O factory
Jul 30 17:00:25.876344 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.876272    2555 factory.go:223] Registration of the crio container factory successfully
Jul 30 17:00:25.876344 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.876333    2555 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 30 17:00:25.876474 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.876353    2555 factory.go:103] Registering Raw factory
Jul 30 17:00:25.876474 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.876380    2555 manager.go:1196] Started watching for new ooms in manager
Jul 30 17:00:25.876895 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.876880    2555 manager.go:349] Starting recovery of all containers
Jul 30 17:00:25.877400 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.877379    2555 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hd6wm"
Jul 30 17:00:25.886174 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:25.886104    2555 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-190.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 30 17:00:25.886561 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:25.886536    2555 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 30 17:00:25.888025 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.887994    2555 manager.go:354] Recovery completed
Jul 30 17:00:25.892620 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.892521    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 30 17:00:25.894650 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.894632    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-190.ec2.internal" event="NodeHasSufficientMemory"
Jul 30 17:00:25.894709 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.894662    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-190.ec2.internal" event="NodeHasNoDiskPressure"
Jul 30 17:00:25.894709 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.894672    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-190.ec2.internal" event="NodeHasSufficientPID"
Jul 30 17:00:25.895179 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.895166    2555 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 30 17:00:25.895179 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.895178    2555 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 30 17:00:25.895268 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.895196    2555 state_mem.go:36] "Initialized new in-memory state store"
Jul 30 17:00:25.895268 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.895208    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 30 17:00:25.895863 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.895847    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-190.ec2.internal" event="NodeHasSufficientMemory"
Jul 30 17:00:25.895923 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.895872    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-190.ec2.internal" event="NodeHasNoDiskPressure"
Jul 30 17:00:25.895923 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.895883    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-190.ec2.internal" event="NodeHasSufficientPID"
Jul 30 17:00:25.896694 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.896681    2555 policy_none.go:49] "None policy: Start"
Jul 30 17:00:25.896756 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.896700    2555 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 30 17:00:25.896756 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.896717    2555 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 30 17:00:25.897631 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.897618    2555 policy_none.go:47] "Start"
Jul 30 17:00:25.930029 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.929994    2555 manager.go:337] "Starting Device Plugin manager"
Jul 30 17:00:25.930140 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:25.930039    2555 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 30 17:00:25.930140 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.930053    2555 server.go:87] "Starting device plugin registration server"
Jul 30 17:00:25.930379 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.930363    2555 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 30 17:00:25.930443 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.930380    2555 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 30 17:00:25.930671 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.930651    2555 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 30 17:00:25.930784 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.930756    2555 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 30 17:00:25.930784 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:25.930767    2555 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 30 17:00:25.931169 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:25.931148    2555 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 30 17:00:25.931248 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:25.931209    2555 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-190.ec2.internal\" not found"
Jul 30 17:00:26.031625 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:26.031591    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 30 17:00:26.032748 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:26.032727    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-190.ec2.internal" event="NodeHasSufficientMemory"
Jul 30 17:00:26.032818 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:26.032762    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-190.ec2.internal" event="NodeHasNoDiskPressure"
Jul 30 17:00:26.032818 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:26.032775    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-190.ec2.internal" event="NodeHasSufficientPID"
Jul 30 17:00:26.032893 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:26.032819    2555 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-190.ec2.internal"
Jul 30 17:00:26.041966 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:26.041946    2555 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-190.ec2.internal"
Jul 30 17:00:26.042077 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:26.041972    2555 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-190.ec2.internal\": node \"ip-10-0-143-190.ec2.internal\" not found"
Jul 30 17:00:26.052924 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:26.052865    2555 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 30 17:00:26.054121 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:26.054100    2555 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 30 17:00:26.054254 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:26.054136    2555 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 30 17:00:26.054254 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:26.054158    2555 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 30 17:00:26.054254 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:26.054206    2555 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 30 17:00:26.055849 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:26.055832    2555 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 30 17:00:26.056740 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:26.056720    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-190.ec2.internal\" not found"
Jul 30 17:00:26.154967 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:26.154909    2555 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-143-190.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-190.ec2.internal"]
Jul 30 17:00:26.155162 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:26.155009    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 30 17:00:26.157008 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:26.156986    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-190.ec2.internal\" not found"
Jul 30 17:00:26.157519 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:26.157504    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-190.ec2.internal" event="NodeHasSufficientMemory"
Jul 30 17:00:26.157604 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:26.157546    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-190.ec2.internal" event="NodeHasNoDiskPressure"
Jul 30 17:00:26.157604 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:26.157558    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-190.ec2.internal" event="NodeHasSufficientPID"
Jul 30 17:00:26.158915 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:26.158903    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 30 17:00:26.159109 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:26.159095    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-190.ec2.internal"
Jul 30 17:00:26.159157 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:26.159123    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 30 17:00:26.159652 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:26.159628    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-190.ec2.internal" event="NodeHasSufficientMemory"
Jul 30 17:00:26.159652 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:26.159641    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-190.ec2.internal" event="NodeHasSufficientMemory"
Jul 30 17:00:26.159791 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:26.159675    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-190.ec2.internal" event="NodeHasNoDiskPressure"
Jul 30 17:00:26.159791 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:26.159688    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-190.ec2.internal" event="NodeHasNoDiskPressure"
Jul 30 17:00:26.159791 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:26.159716    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-190.ec2.internal" event="NodeHasSufficientPID"
Jul 30 17:00:26.159791 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:26.159690    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-190.ec2.internal" event="NodeHasSufficientPID"
Jul 30 17:00:26.160851 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:26.160835    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-190.ec2.internal"
Jul 30 17:00:26.160909 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:26.160889    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 30 17:00:26.161620 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:26.161602    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-190.ec2.internal" event="NodeHasSufficientMemory"
Jul 30 17:00:26.161720 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:26.161647    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-190.ec2.internal" event="NodeHasNoDiskPressure"
Jul 30 17:00:26.161720 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:26.161661    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-190.ec2.internal" event="NodeHasSufficientPID"
Jul 30 17:00:26.179189 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:26.179168    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/041d73f2cda3ec42dcee6249e2df6511-config\") pod \"kube-apiserver-proxy-ip-10-0-143-190.ec2.internal\" (UID: \"041d73f2cda3ec42dcee6249e2df6511\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-190.ec2.internal"
Jul 30 17:00:26.179294 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:26.179194    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3e02b1d2169088ae669788950ba2bc8f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-190.ec2.internal\" (UID: \"3e02b1d2169088ae669788950ba2bc8f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-190.ec2.internal"
Jul 30 17:00:26.179294 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:26.179223    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e02b1d2169088ae669788950ba2bc8f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-190.ec2.internal\" (UID: \"3e02b1d2169088ae669788950ba2bc8f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-190.ec2.internal"
Jul 30 17:00:26.185689 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:26.185674    2555 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-190.ec2.internal\" not found" node="ip-10-0-143-190.ec2.internal"
Jul 30 17:00:26.189297 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:26.189280    2555 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-190.ec2.internal\" not found" node="ip-10-0-143-190.ec2.internal"
Jul 30 17:00:26.257674 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:26.257650    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-190.ec2.internal\" not found"
Jul 30 17:00:26.279548 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:26.279517    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/041d73f2cda3ec42dcee6249e2df6511-config\") pod \"kube-apiserver-proxy-ip-10-0-143-190.ec2.internal\" (UID: \"041d73f2cda3ec42dcee6249e2df6511\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-190.ec2.internal"
Jul 30 17:00:26.279627 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:26.279552    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3e02b1d2169088ae669788950ba2bc8f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-190.ec2.internal\" (UID: \"3e02b1d2169088ae669788950ba2bc8f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-190.ec2.internal"
Jul 30 17:00:26.279627 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:26.279572    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e02b1d2169088ae669788950ba2bc8f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-190.ec2.internal\" (UID: \"3e02b1d2169088ae669788950ba2bc8f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-190.ec2.internal"
Jul 30 17:00:26.279710 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:26.279622    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3e02b1d2169088ae669788950ba2bc8f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-190.ec2.internal\" (UID: \"3e02b1d2169088ae669788950ba2bc8f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-190.ec2.internal"
Jul 30 17:00:26.279710 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:26.279642    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/041d73f2cda3ec42dcee6249e2df6511-config\") pod \"kube-apiserver-proxy-ip-10-0-143-190.ec2.internal\" (UID: \"041d73f2cda3ec42dcee6249e2df6511\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-190.ec2.internal"
Jul 30 17:00:26.279710 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:26.279680    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e02b1d2169088ae669788950ba2bc8f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-190.ec2.internal\" (UID: \"3e02b1d2169088ae669788950ba2bc8f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-190.ec2.internal"
Jul 30 17:00:26.359982 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:26.359893    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-190.ec2.internal\" not found"
Jul 30 17:00:26.463580 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:26.463535    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-190.ec2.internal\" not found"
Jul 30 17:00:26.488887 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:26.488857    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-190.ec2.internal"
Jul 30 17:00:26.492999 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:26.492976    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-190.ec2.internal"
Jul 30 17:00:26.563876 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:26.563836    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-190.ec2.internal\" not found"
Jul 30 17:00:26.669275 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:26.669197    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-190.ec2.internal\" not found"
Jul 30 17:00:26.771024 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:26.770961    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-190.ec2.internal\" not found"
Jul 30 17:00:26.822348 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:26.822315    2555 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 30 17:00:26.822896 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:26.822493    2555 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 30 17:00:26.871379 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:26.871347    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-190.ec2.internal\" not found"
Jul 30 17:00:26.882997 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:26.882940    2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-29 16:55:26 +0000 UTC" deadline="2028-02-12 22:04:52.092639758 +0000 UTC"
Jul 30 17:00:26.882997 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:26.882991    2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13493h4m25.209653484s"
Jul 30 17:00:26.884793 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:26.884772    2555 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 30 17:00:26.972322 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:26.972295    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-190.ec2.internal\" not found"
Jul 30 17:00:27.072109 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:27.072071    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod041d73f2cda3ec42dcee6249e2df6511.slice/crio-e386cd85406b710c928e3d4e0918530bf107137ddfbe27213c922c9a6f39cdcb WatchSource:0}: Error finding container e386cd85406b710c928e3d4e0918530bf107137ddfbe27213c922c9a6f39cdcb: Status 404 returned error can't find the container with id e386cd85406b710c928e3d4e0918530bf107137ddfbe27213c922c9a6f39cdcb
Jul 30 17:00:27.072380 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:27.072359    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-190.ec2.internal\" not found"
Jul 30 17:00:27.072537 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:27.072509    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e02b1d2169088ae669788950ba2bc8f.slice/crio-b36841d9d138eca2dab1a9e62a6a043a581122afb4fb0c722026052f18a1ef89 WatchSource:0}: Error finding container b36841d9d138eca2dab1a9e62a6a043a581122afb4fb0c722026052f18a1ef89: Status 404 returned error can't find the container with id b36841d9d138eca2dab1a9e62a6a043a581122afb4fb0c722026052f18a1ef89
Jul 30 17:00:27.078796 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.078780    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 30 17:00:27.175154 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:27.175111    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-190.ec2.internal\" not found"
Jul 30 17:00:27.277530 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:27.277460    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-190.ec2.internal\" not found"
Jul 30 17:00:27.307230 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.307203    2555 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 30 17:00:27.374375 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.374344    2555 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 30 17:00:27.376315 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.376292    2555 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-190.ec2.internal"
Jul 30 17:00:27.387524 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.387502    2555 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 30 17:00:27.388599 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.388586    2555 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-190.ec2.internal"
Jul 30 17:00:27.404845 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.404825    2555 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 30 17:00:27.859087 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.859057    2555 apiserver.go:52] "Watching apiserver"
Jul 30 17:00:27.867432 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.867407    2555 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 30 17:00:27.868643 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.868620    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-qbp2g","openshift-multus/multus-nfpq4","openshift-network-diagnostics/network-check-target-qvgrv","kube-system/global-pull-secret-syncer-9hlj5","openshift-image-registry/node-ca-xglts","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-190.ec2.internal","openshift-multus/network-metrics-daemon-55hfh","openshift-network-operator/iptables-alerter-2tdqf","openshift-ovn-kubernetes/ovnkube-node-t87pw","kube-system/konnectivity-agent-8lckj","kube-system/kube-apiserver-proxy-ip-10-0-143-190.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9hmcr","openshift-cluster-node-tuning-operator/tuned-bncwx","openshift-dns/node-resolver-97k9j"]
Jul 30 17:00:27.870896 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.870865    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-55hfh"
Jul 30 17:00:27.871008 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:27.870958    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-55hfh" podUID="e85f9d71-fe3d-49d8-93e2-e4ac101603c7"
Jul 30 17:00:27.872831 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.872810    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bncwx"
Jul 30 17:00:27.875061 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.874925    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvgrv"
Jul 30 17:00:27.875061 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:27.874999    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvgrv" podUID="fc0a3112-260e-47c8-87fd-bf3666ea17d0"
Jul 30 17:00:27.875403 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.875217    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-xgpz7\""
Jul 30 17:00:27.875498 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.875457    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 30 17:00:27.875560 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.875492    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 30 17:00:27.875664 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.875618    2555 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 30 17:00:27.877246 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.877180    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hlj5"
Jul 30 17:00:27.877357 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:27.877248    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9hlj5" podUID="51afa6bb-724f-47ab-b47b-36dc8a61ca25"
Jul 30 17:00:27.879303 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.879288    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xglts"
Jul 30 17:00:27.881560 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.881539    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-97k9j"
Jul 30 17:00:27.882296 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.882155    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 30 17:00:27.882500 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.882480    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 30 17:00:27.882581 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.882556    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 30 17:00:27.882808 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.882788    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-d724j\""
Jul 30 17:00:27.883938 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.883919    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-t87pw"
Jul 30 17:00:27.884437 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.884412    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 30 17:00:27.884593 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.884572    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-6pkl2\""
Jul 30 17:00:27.885078 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.885055    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 30 17:00:27.886476 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.886455    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qbp2g"
Jul 30 17:00:27.886570 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.886515    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-rjgv6\""
Jul 30 17:00:27.886627 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.886605    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 30 17:00:27.887145 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.887126    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 30 17:00:27.888454 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.888429    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 30 17:00:27.888604 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.888584    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrkrk\" (UniqueName: \"kubernetes.io/projected/e85f9d71-fe3d-49d8-93e2-e4ac101603c7-kube-api-access-jrkrk\") pod \"network-metrics-daemon-55hfh\" (UID: \"e85f9d71-fe3d-49d8-93e2-e4ac101603c7\") " pod="openshift-multus/network-metrics-daemon-55hfh"
Jul 30 17:00:27.888678 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.888628    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/80fefbe4-044f-4c09-b4ae-4bcc3d5684d6-tmp\") pod \"tuned-bncwx\" (UID: \"80fefbe4-044f-4c09-b4ae-4bcc3d5684d6\") " pod="openshift-cluster-node-tuning-operator/tuned-bncwx"
Jul 30 17:00:27.888678 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.888670    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/51afa6bb-724f-47ab-b47b-36dc8a61ca25-dbus\") pod \"global-pull-secret-syncer-9hlj5\" (UID: \"51afa6bb-724f-47ab-b47b-36dc8a61ca25\") " pod="kube-system/global-pull-secret-syncer-9hlj5"
Jul 30 17:00:27.888779 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.888699    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e85f9d71-fe3d-49d8-93e2-e4ac101603c7-metrics-certs\") pod \"network-metrics-daemon-55hfh\" (UID: \"e85f9d71-fe3d-49d8-93e2-e4ac101603c7\") " pod="openshift-multus/network-metrics-daemon-55hfh"
Jul 30 17:00:27.888779 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.888709    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 30 17:00:27.888779 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.888736    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/80fefbe4-044f-4c09-b4ae-4bcc3d5684d6-etc-sysctl-d\") pod \"tuned-bncwx\" (UID: \"80fefbe4-044f-4c09-b4ae-4bcc3d5684d6\") " pod="openshift-cluster-node-tuning-operator/tuned-bncwx"
Jul 30 17:00:27.888922 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.888791    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 30 17:00:27.888972 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.888795    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/80fefbe4-044f-4c09-b4ae-4bcc3d5684d6-var-lib-kubelet\") pod \"tuned-bncwx\" (UID: \"80fefbe4-044f-4c09-b4ae-4bcc3d5684d6\") " pod="openshift-cluster-node-tuning-operator/tuned-bncwx"
Jul 30 17:00:27.889020 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.888987    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/80fefbe4-044f-4c09-b4ae-4bcc3d5684d6-etc-sysconfig\") pod \"tuned-bncwx\" (UID: \"80fefbe4-044f-4c09-b4ae-4bcc3d5684d6\") " pod="openshift-cluster-node-tuning-operator/tuned-bncwx"
Jul 30 17:00:27.889071 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.889015    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/80fefbe4-044f-4c09-b4ae-4bcc3d5684d6-etc-systemd\") pod \"tuned-bncwx\" (UID: \"80fefbe4-044f-4c09-b4ae-4bcc3d5684d6\") " pod="openshift-cluster-node-tuning-operator/tuned-bncwx"
Jul 30 17:00:27.889148 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.889047    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/80fefbe4-044f-4c09-b4ae-4bcc3d5684d6-sys\") pod \"tuned-bncwx\" (UID: \"80fefbe4-044f-4c09-b4ae-4bcc3d5684d6\") " pod="openshift-cluster-node-tuning-operator/tuned-bncwx"
Jul 30 17:00:27.889148 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.889077    2555 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 30 17:00:27.889148 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.889101    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/80fefbe4-044f-4c09-b4ae-4bcc3d5684d6-lib-modules\") pod \"tuned-bncwx\" (UID: \"80fefbe4-044f-4c09-b4ae-4bcc3d5684d6\") " pod="openshift-cluster-node-tuning-operator/tuned-bncwx"
Jul 30 17:00:27.889904 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.889695    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 30 17:00:27.889904 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.889720    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/80fefbe4-044f-4c09-b4ae-4bcc3d5684d6-host\") pod \"tuned-bncwx\" (UID: \"80fefbe4-044f-4c09-b4ae-4bcc3d5684d6\") " pod="openshift-cluster-node-tuning-operator/tuned-bncwx"
Jul 30 17:00:27.889904 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.889772    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29sfv\" (UniqueName: \"kubernetes.io/projected/80fefbe4-044f-4c09-b4ae-4bcc3d5684d6-kube-api-access-29sfv\") pod \"tuned-bncwx\" (UID: \"80fefbe4-044f-4c09-b4ae-4bcc3d5684d6\") " pod="openshift-cluster-node-tuning-operator/tuned-bncwx"
Jul 30 17:00:27.889904 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.889809    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/80fefbe4-044f-4c09-b4ae-4bcc3d5684d6-etc-modprobe-d\") pod \"tuned-bncwx\" (UID: \"80fefbe4-044f-4c09-b4ae-4bcc3d5684d6\") " pod="openshift-cluster-node-tuning-operator/tuned-bncwx"
Jul 30 17:00:27.889904 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.889837    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/80fefbe4-044f-4c09-b4ae-4bcc3d5684d6-etc-kubernetes\") pod \"tuned-bncwx\" (UID: \"80fefbe4-044f-4c09-b4ae-4bcc3d5684d6\") " pod="openshift-cluster-node-tuning-operator/tuned-bncwx"
Jul 30 17:00:27.890248 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.889939    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/80fefbe4-044f-4c09-b4ae-4bcc3d5684d6-etc-sysctl-conf\") pod \"tuned-bncwx\" (UID: \"80fefbe4-044f-4c09-b4ae-4bcc3d5684d6\") " pod="openshift-cluster-node-tuning-operator/tuned-bncwx"
Jul 30 17:00:27.890248 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.889944    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nfpq4"
Jul 30 17:00:27.890248 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.890026    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/80fefbe4-044f-4c09-b4ae-4bcc3d5684d6-run\") pod \"tuned-bncwx\" (UID: \"80fefbe4-044f-4c09-b4ae-4bcc3d5684d6\") " pod="openshift-cluster-node-tuning-operator/tuned-bncwx"
Jul 30 17:00:27.890248 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.890064    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/80fefbe4-044f-4c09-b4ae-4bcc3d5684d6-etc-tuned\") pod \"tuned-bncwx\" (UID: \"80fefbe4-044f-4c09-b4ae-4bcc3d5684d6\") " pod="openshift-cluster-node-tuning-operator/tuned-bncwx"
Jul 30 17:00:27.890248 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.890098    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmqln\" (UniqueName: \"kubernetes.io/projected/fc0a3112-260e-47c8-87fd-bf3666ea17d0-kube-api-access-rmqln\") pod \"network-check-target-qvgrv\" (UID: \"fc0a3112-260e-47c8-87fd-bf3666ea17d0\") " pod="openshift-network-diagnostics/network-check-target-qvgrv"
Jul 30 17:00:27.890248 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.890134    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/51afa6bb-724f-47ab-b47b-36dc8a61ca25-kubelet-config\") pod \"global-pull-secret-syncer-9hlj5\" (UID: \"51afa6bb-724f-47ab-b47b-36dc8a61ca25\") " pod="kube-system/global-pull-secret-syncer-9hlj5"
Jul 30 17:00:27.890248 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.890190    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/51afa6bb-724f-47ab-b47b-36dc8a61ca25-original-pull-secret\") pod \"global-pull-secret-syncer-9hlj5\" (UID: \"51afa6bb-724f-47ab-b47b-36dc8a61ca25\") " pod="kube-system/global-pull-secret-syncer-9hlj5"
Jul 30 17:00:27.890603 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.890265    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 30 17:00:27.890603 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.890358    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-bbpp2\""
Jul 30 17:00:27.890603 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.890509    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 30 17:00:27.890603 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.890594    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 30 17:00:27.890805 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.890760    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 30 17:00:27.891683 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.891661    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 30 17:00:27.892480 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.892460    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8lckj"
Jul 30 17:00:27.893052 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.893015    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-58qzt\""
Jul 30 17:00:27.893846 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.893827    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 30 17:00:27.894768 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.894750    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 30 17:00:27.894899 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.894880    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 30 17:00:27.895074 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.895032    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-vqrh7\""
Jul 30 17:00:27.895505 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.895484    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2tdqf"
Jul 30 17:00:27.898633 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.897976    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9hmcr"
Jul 30 17:00:27.898633 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.898084    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-2ljqx\""
Jul 30 17:00:27.898633 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.898325    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 30 17:00:27.898633 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.898374    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 30 17:00:27.898633 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.898470    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 30 17:00:27.900525 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.900467    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-kwpcb\""
Jul 30 17:00:27.900703 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.900683    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 30 17:00:27.900703 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.900694    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 30 17:00:27.900841 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.900687    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 30 17:00:27.923158 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.923131    2555 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-6dsfp"
Jul 30 17:00:27.931336 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.931315    2555 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-6dsfp"
Jul 30 17:00:27.976906 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.976874    2555 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 30 17:00:27.991247 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.991219    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/80fefbe4-044f-4c09-b4ae-4bcc3d5684d6-sys\") pod \"tuned-bncwx\" (UID: \"80fefbe4-044f-4c09-b4ae-4bcc3d5684d6\") " pod="openshift-cluster-node-tuning-operator/tuned-bncwx"
Jul 30 17:00:27.991422 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.991263    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2093549-0eca-49bd-96d3-2e0c01a4c19c-etc-kubernetes\") pod \"multus-nfpq4\" (UID: \"a2093549-0eca-49bd-96d3-2e0c01a4c19c\") " pod="openshift-multus/multus-nfpq4"
Jul 30 17:00:27.991422 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.991290    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/80fefbe4-044f-4c09-b4ae-4bcc3d5684d6-etc-kubernetes\") pod \"tuned-bncwx\" (UID: \"80fefbe4-044f-4c09-b4ae-4bcc3d5684d6\") " pod="openshift-cluster-node-tuning-operator/tuned-bncwx"
Jul 30 17:00:27.991422 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.991335    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfkrg\" (UniqueName: \"kubernetes.io/projected/a28bc718-4b4f-408b-b1ca-57f116295c99-kube-api-access-mfkrg\") pod \"aws-ebs-csi-driver-node-9hmcr\" (UID: \"a28bc718-4b4f-408b-b1ca-57f116295c99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9hmcr"
Jul 30 17:00:27.991422 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.991358    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/80fefbe4-044f-4c09-b4ae-4bcc3d5684d6-etc-kubernetes\") pod \"tuned-bncwx\" (UID: \"80fefbe4-044f-4c09-b4ae-4bcc3d5684d6\") " pod="openshift-cluster-node-tuning-operator/tuned-bncwx"
Jul 30 17:00:27.991422 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.991374    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1dcce1f-b59e-47cc-9015-429f8a7858cd-etc-openvswitch\") pod \"ovnkube-node-t87pw\" (UID: \"e1dcce1f-b59e-47cc-9015-429f8a7858cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-t87pw"
Jul 30 17:00:27.991422 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.991403    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e1dcce1f-b59e-47cc-9015-429f8a7858cd-node-log\") pod \"ovnkube-node-t87pw\" (UID: \"e1dcce1f-b59e-47cc-9015-429f8a7858cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-t87pw"
Jul 30 17:00:27.991422 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.991359    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/80fefbe4-044f-4c09-b4ae-4bcc3d5684d6-sys\") pod \"tuned-bncwx\" (UID: \"80fefbe4-044f-4c09-b4ae-4bcc3d5684d6\") " pod="openshift-cluster-node-tuning-operator/tuned-bncwx"
Jul 30 17:00:27.991750 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.991424    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/97f5f5b8-ce32-41cd-9b5c-c94f2ae6bbfe-serviceca\") pod \"node-ca-xglts\" (UID: \"97f5f5b8-ce32-41cd-9b5c-c94f2ae6bbfe\") " pod="openshift-image-registry/node-ca-xglts"
Jul 30 17:00:27.991750 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.991462    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a2093549-0eca-49bd-96d3-2e0c01a4c19c-hostroot\") pod \"multus-nfpq4\" (UID: \"a2093549-0eca-49bd-96d3-2e0c01a4c19c\") " pod="openshift-multus/multus-nfpq4"
Jul 30 17:00:27.991750 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.991502    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a2093549-0eca-49bd-96d3-2e0c01a4c19c-multus-daemon-config\") pod \"multus-nfpq4\" (UID: \"a2093549-0eca-49bd-96d3-2e0c01a4c19c\") " pod="openshift-multus/multus-nfpq4"
Jul 30 17:00:27.991750 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.991524    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3534e135-e217-49b8-8866-8cd33104b77e-hosts-file\") pod \"node-resolver-97k9j\" (UID: \"3534e135-e217-49b8-8866-8cd33104b77e\") " pod="openshift-dns/node-resolver-97k9j"
Jul 30 17:00:27.991750 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.991571    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgmjc\" (UniqueName: \"kubernetes.io/projected/3534e135-e217-49b8-8866-8cd33104b77e-kube-api-access-kgmjc\") pod \"node-resolver-97k9j\" (UID: \"3534e135-e217-49b8-8866-8cd33104b77e\") " pod="openshift-dns/node-resolver-97k9j"
Jul 30 17:00:27.991750 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.991611    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9fcdb3a5-74bc-44be-993f-fbc90e5db202-cni-binary-copy\") pod \"multus-additional-cni-plugins-qbp2g\" (UID: \"9fcdb3a5-74bc-44be-993f-fbc90e5db202\") " pod="openshift-multus/multus-additional-cni-plugins-qbp2g"
Jul 30 17:00:27.991750 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.991655    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/51afa6bb-724f-47ab-b47b-36dc8a61ca25-dbus\") pod \"global-pull-secret-syncer-9hlj5\" (UID: \"51afa6bb-724f-47ab-b47b-36dc8a61ca25\") " pod="kube-system/global-pull-secret-syncer-9hlj5"
Jul 30 17:00:27.991750 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.991687    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e1dcce1f-b59e-47cc-9015-429f8a7858cd-host-run-netns\") pod \"ovnkube-node-t87pw\" (UID: \"e1dcce1f-b59e-47cc-9015-429f8a7858cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-t87pw"
Jul 30 17:00:27.992166 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.991751    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e1dcce1f-b59e-47cc-9015-429f8a7858cd-run-ovn\") pod \"ovnkube-node-t87pw\" (UID: \"e1dcce1f-b59e-47cc-9015-429f8a7858cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-t87pw"
Jul 30 17:00:27.992166 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.991801    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e1dcce1f-b59e-47cc-9015-429f8a7858cd-log-socket\") pod \"ovnkube-node-t87pw\" (UID: \"e1dcce1f-b59e-47cc-9015-429f8a7858cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-t87pw"
Jul 30 17:00:27.992166 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.991839    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e1dcce1f-b59e-47cc-9015-429f8a7858cd-ovnkube-script-lib\") pod \"ovnkube-node-t87pw\" (UID: \"e1dcce1f-b59e-47cc-9015-429f8a7858cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-t87pw"
Jul 30 17:00:27.992166 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.991866    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a28bc718-4b4f-408b-b1ca-57f116295c99-socket-dir\") pod \"aws-ebs-csi-driver-node-9hmcr\" (UID: \"a28bc718-4b4f-408b-b1ca-57f116295c99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9hmcr"
Jul 30 17:00:27.992166 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.991893    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/80fefbe4-044f-4c09-b4ae-4bcc3d5684d6-host\") pod \"tuned-bncwx\" (UID: \"80fefbe4-044f-4c09-b4ae-4bcc3d5684d6\") " pod="openshift-cluster-node-tuning-operator/tuned-bncwx"
Jul 30 17:00:27.992166 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.991925    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29sfv\" (UniqueName: \"kubernetes.io/projected/80fefbe4-044f-4c09-b4ae-4bcc3d5684d6-kube-api-access-29sfv\") pod \"tuned-bncwx\" (UID: \"80fefbe4-044f-4c09-b4ae-4bcc3d5684d6\") " pod="openshift-cluster-node-tuning-operator/tuned-bncwx"
Jul 30 17:00:27.992166 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.991951    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e1dcce1f-b59e-47cc-9015-429f8a7858cd-systemd-units\") pod \"ovnkube-node-t87pw\" (UID: \"e1dcce1f-b59e-47cc-9015-429f8a7858cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-t87pw"
Jul 30 17:00:27.992166 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.991975    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1dcce1f-b59e-47cc-9015-429f8a7858cd-var-lib-openvswitch\") pod \"ovnkube-node-t87pw\" (UID: \"e1dcce1f-b59e-47cc-9015-429f8a7858cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-t87pw"
Jul 30 17:00:27.992166 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.991981    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/80fefbe4-044f-4c09-b4ae-4bcc3d5684d6-host\") pod \"tuned-bncwx\" (UID: \"80fefbe4-044f-4c09-b4ae-4bcc3d5684d6\") " pod="openshift-cluster-node-tuning-operator/tuned-bncwx"
Jul 30 17:00:27.992166 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.991922    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/51afa6bb-724f-47ab-b47b-36dc8a61ca25-dbus\") pod \"global-pull-secret-syncer-9hlj5\" (UID: \"51afa6bb-724f-47ab-b47b-36dc8a61ca25\") " pod="kube-system/global-pull-secret-syncer-9hlj5"
Jul 30 17:00:27.992166 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.992001    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a2093549-0eca-49bd-96d3-2e0c01a4c19c-host-run-k8s-cni-cncf-io\") pod \"multus-nfpq4\" (UID: \"a2093549-0eca-49bd-96d3-2e0c01a4c19c\") " pod="openshift-multus/multus-nfpq4"
Jul 30 17:00:27.992166 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.992061    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a28bc718-4b4f-408b-b1ca-57f116295c99-device-dir\") pod \"aws-ebs-csi-driver-node-9hmcr\" (UID: \"a28bc718-4b4f-408b-b1ca-57f116295c99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9hmcr"
Jul 30 17:00:27.992166 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.992097    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1dcce1f-b59e-47cc-9015-429f8a7858cd-host-run-ovn-kubernetes\") pod \"ovnkube-node-t87pw\" (UID: \"e1dcce1f-b59e-47cc-9015-429f8a7858cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-t87pw"
Jul 30 17:00:27.992166 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.992125    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ed0f2ced-c500-4288-ac35-3e5599c9d25f-iptables-alerter-script\") pod \"iptables-alerter-2tdqf\" (UID: \"ed0f2ced-c500-4288-ac35-3e5599c9d25f\") " pod="openshift-network-operator/iptables-alerter-2tdqf"
Jul 30 17:00:27.992166 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.992150    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9fcdb3a5-74bc-44be-993f-fbc90e5db202-os-release\") pod \"multus-additional-cni-plugins-qbp2g\" (UID: \"9fcdb3a5-74bc-44be-993f-fbc90e5db202\") " pod="openshift-multus/multus-additional-cni-plugins-qbp2g"
Jul 30 17:00:27.992854 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.992181    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2093549-0eca-49bd-96d3-2e0c01a4c19c-system-cni-dir\") pod \"multus-nfpq4\" (UID: \"a2093549-0eca-49bd-96d3-2e0c01a4c19c\") " pod="openshift-multus/multus-nfpq4"
Jul 30 17:00:27.992854 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.992207    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a2093549-0eca-49bd-96d3-2e0c01a4c19c-host-var-lib-cni-bin\") pod \"multus-nfpq4\" (UID: \"a2093549-0eca-49bd-96d3-2e0c01a4c19c\") " pod="openshift-multus/multus-nfpq4"
Jul 30 17:00:27.992854 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.992251    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2093549-0eca-49bd-96d3-2e0c01a4c19c-host-var-lib-kubelet\") pod \"multus-nfpq4\" (UID: \"a2093549-0eca-49bd-96d3-2e0c01a4c19c\") " pod="openshift-multus/multus-nfpq4"
Jul 30 17:00:27.992854 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.992280    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/51afa6bb-724f-47ab-b47b-36dc8a61ca25-kubelet-config\") pod \"global-pull-secret-syncer-9hlj5\" (UID: \"51afa6bb-724f-47ab-b47b-36dc8a61ca25\") " pod="kube-system/global-pull-secret-syncer-9hlj5"
Jul 30 17:00:27.992854 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.992306    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/80fefbe4-044f-4c09-b4ae-4bcc3d5684d6-tmp\") pod \"tuned-bncwx\" (UID: \"80fefbe4-044f-4c09-b4ae-4bcc3d5684d6\") " pod="openshift-cluster-node-tuning-operator/tuned-bncwx"
Jul 30 17:00:27.992854 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.992334    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e1dcce1f-b59e-47cc-9015-429f8a7858cd-ovnkube-config\") pod \"ovnkube-node-t87pw\" (UID: \"e1dcce1f-b59e-47cc-9015-429f8a7858cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-t87pw"
Jul 30 17:00:27.992854 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.992349    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/51afa6bb-724f-47ab-b47b-36dc8a61ca25-kubelet-config\") pod \"global-pull-secret-syncer-9hlj5\" (UID: \"51afa6bb-724f-47ab-b47b-36dc8a61ca25\") " pod="kube-system/global-pull-secret-syncer-9hlj5"
Jul 30 17:00:27.992854 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.992381    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e1dcce1f-b59e-47cc-9015-429f8a7858cd-env-overrides\") pod \"ovnkube-node-t87pw\" (UID: \"e1dcce1f-b59e-47cc-9015-429f8a7858cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-t87pw"
Jul 30 17:00:27.992854 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.992409    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bzxp\" (UniqueName: \"kubernetes.io/projected/e1dcce1f-b59e-47cc-9015-429f8a7858cd-kube-api-access-6bzxp\") pod \"ovnkube-node-t87pw\" (UID: \"e1dcce1f-b59e-47cc-9015-429f8a7858cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-t87pw"
Jul 30 17:00:27.992854 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.992435    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9fcdb3a5-74bc-44be-993f-fbc90e5db202-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qbp2g\" (UID: \"9fcdb3a5-74bc-44be-993f-fbc90e5db202\") " pod="openshift-multus/multus-additional-cni-plugins-qbp2g"
Jul 30 17:00:27.992854 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.992459    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a2093549-0eca-49bd-96d3-2e0c01a4c19c-host-run-netns\") pod \"multus-nfpq4\" (UID: \"a2093549-0eca-49bd-96d3-2e0c01a4c19c\") " pod="openshift-multus/multus-nfpq4"
Jul 30 17:00:27.992854 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.992481    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd6ws\" (UniqueName: \"kubernetes.io/projected/a2093549-0eca-49bd-96d3-2e0c01a4c19c-kube-api-access-nd6ws\") pod \"multus-nfpq4\" (UID: \"a2093549-0eca-49bd-96d3-2e0c01a4c19c\") " pod="openshift-multus/multus-nfpq4"
Jul 30 17:00:27.992854 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.992499    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/80fefbe4-044f-4c09-b4ae-4bcc3d5684d6-etc-sysctl-d\") pod \"tuned-bncwx\" (UID: \"80fefbe4-044f-4c09-b4ae-4bcc3d5684d6\") " pod="openshift-cluster-node-tuning-operator/tuned-bncwx"
Jul 30 17:00:27.992854 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.992562    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ed0f2ced-c500-4288-ac35-3e5599c9d25f-host-slash\") pod \"iptables-alerter-2tdqf\" (UID: \"ed0f2ced-c500-4288-ac35-3e5599c9d25f\") " pod="openshift-network-operator/iptables-alerter-2tdqf"
Jul 30 17:00:27.992854 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.992604    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/97f5f5b8-ce32-41cd-9b5c-c94f2ae6bbfe-host\") pod \"node-ca-xglts\" (UID: \"97f5f5b8-ce32-41cd-9b5c-c94f2ae6bbfe\") " pod="openshift-image-registry/node-ca-xglts"
Jul 30 17:00:27.992854 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.992626    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv555\" (UniqueName: \"kubernetes.io/projected/97f5f5b8-ce32-41cd-9b5c-c94f2ae6bbfe-kube-api-access-vv555\") pod \"node-ca-xglts\" (UID: \"97f5f5b8-ce32-41cd-9b5c-c94f2ae6bbfe\") " pod="openshift-image-registry/node-ca-xglts"
Jul 30 17:00:27.992854 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.992662    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhhj5\" (UniqueName: \"kubernetes.io/projected/9fcdb3a5-74bc-44be-993f-fbc90e5db202-kube-api-access-mhhj5\") pod \"multus-additional-cni-plugins-qbp2g\" (UID: \"9fcdb3a5-74bc-44be-993f-fbc90e5db202\") " pod="openshift-multus/multus-additional-cni-plugins-qbp2g"
Jul 30 17:00:27.993691 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.992699    2555 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 30 17:00:27.993691 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.992709    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/80fefbe4-044f-4c09-b4ae-4bcc3d5684d6-etc-sysctl-d\") pod \"tuned-bncwx\" (UID: \"80fefbe4-044f-4c09-b4ae-4bcc3d5684d6\") " pod="openshift-cluster-node-tuning-operator/tuned-bncwx"
Jul 30 17:00:27.993691 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.992755    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a2093549-0eca-49bd-96d3-2e0c01a4c19c-cnibin\") pod \"multus-nfpq4\" (UID: \"a2093549-0eca-49bd-96d3-2e0c01a4c19c\") " pod="openshift-multus/multus-nfpq4"
Jul 30 17:00:27.993691 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.992814    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a2093549-0eca-49bd-96d3-2e0c01a4c19c-multus-socket-dir-parent\") pod \"multus-nfpq4\" (UID: \"a2093549-0eca-49bd-96d3-2e0c01a4c19c\") " pod="openshift-multus/multus-nfpq4"
Jul 30 17:00:27.993691 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.992841    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/80fefbe4-044f-4c09-b4ae-4bcc3d5684d6-etc-sysconfig\") pod \"tuned-bncwx\" (UID: \"80fefbe4-044f-4c09-b4ae-4bcc3d5684d6\") " pod="openshift-cluster-node-tuning-operator/tuned-bncwx"
Jul 30 17:00:27.993691 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.992865    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a2093549-0eca-49bd-96d3-2e0c01a4c19c-multus-conf-dir\") pod \"multus-nfpq4\" (UID: \"a2093549-0eca-49bd-96d3-2e0c01a4c19c\") " pod="openshift-multus/multus-nfpq4"
Jul 30 17:00:27.993691 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.992888    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1dcce1f-b59e-47cc-9015-429f8a7858cd-host-kubelet\") pod \"ovnkube-node-t87pw\" (UID: \"e1dcce1f-b59e-47cc-9015-429f8a7858cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-t87pw"
Jul 30 17:00:27.993691 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.992930    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/80fefbe4-044f-4c09-b4ae-4bcc3d5684d6-etc-sysconfig\") pod \"tuned-bncwx\" (UID: \"80fefbe4-044f-4c09-b4ae-4bcc3d5684d6\") " pod="openshift-cluster-node-tuning-operator/tuned-bncwx"
Jul 30 17:00:27.993691 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.992945    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e1dcce1f-b59e-47cc-9015-429f8a7858cd-run-systemd\") pod \"ovnkube-node-t87pw\" (UID: \"e1dcce1f-b59e-47cc-9015-429f8a7858cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-t87pw"
Jul 30 17:00:27.993691 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.992995    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3534e135-e217-49b8-8866-8cd33104b77e-tmp-dir\") pod \"node-resolver-97k9j\" (UID: \"3534e135-e217-49b8-8866-8cd33104b77e\") " pod="openshift-dns/node-resolver-97k9j"
Jul 30 17:00:27.993691 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.993054    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9fcdb3a5-74bc-44be-993f-fbc90e5db202-system-cni-dir\") pod \"multus-additional-cni-plugins-qbp2g\" (UID: \"9fcdb3a5-74bc-44be-993f-fbc90e5db202\") " pod="openshift-multus/multus-additional-cni-plugins-qbp2g"
Jul 30 17:00:27.993691 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.993103    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmqln\" (UniqueName: \"kubernetes.io/projected/fc0a3112-260e-47c8-87fd-bf3666ea17d0-kube-api-access-rmqln\") pod \"network-check-target-qvgrv\" (UID: \"fc0a3112-260e-47c8-87fd-bf3666ea17d0\") " pod="openshift-network-diagnostics/network-check-target-qvgrv"
Jul 30 17:00:27.993691 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.993139    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/51afa6bb-724f-47ab-b47b-36dc8a61ca25-original-pull-secret\") pod \"global-pull-secret-syncer-9hlj5\" (UID: \"51afa6bb-724f-47ab-b47b-36dc8a61ca25\") " pod="kube-system/global-pull-secret-syncer-9hlj5"
Jul 30 17:00:27.993691 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.993166    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrkrk\" (UniqueName: \"kubernetes.io/projected/e85f9d71-fe3d-49d8-93e2-e4ac101603c7-kube-api-access-jrkrk\") pod \"network-metrics-daemon-55hfh\" (UID: \"e85f9d71-fe3d-49d8-93e2-e4ac101603c7\") " pod="openshift-multus/network-metrics-daemon-55hfh"
Jul 30 17:00:27.993691 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.993195    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9485af8e-c7d9-4b8a-ab94-aa3b30415381-konnectivity-ca\") pod \"konnectivity-agent-8lckj\" (UID: \"9485af8e-c7d9-4b8a-ab94-aa3b30415381\") " pod="kube-system/konnectivity-agent-8lckj"
Jul 30 17:00:27.993691 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.993227    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9fcdb3a5-74bc-44be-993f-fbc90e5db202-cnibin\") pod \"multus-additional-cni-plugins-qbp2g\" (UID: \"9fcdb3a5-74bc-44be-993f-fbc90e5db202\") " pod="openshift-multus/multus-additional-cni-plugins-qbp2g"
Jul 30 17:00:27.993691 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.993272    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9fcdb3a5-74bc-44be-993f-fbc90e5db202-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qbp2g\" (UID: \"9fcdb3a5-74bc-44be-993f-fbc90e5db202\") " pod="openshift-multus/multus-additional-cni-plugins-qbp2g"
Jul 30 17:00:27.994443 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:27.993276    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 30 17:00:27.994443 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.993408    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9fcdb3a5-74bc-44be-993f-fbc90e5db202-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qbp2g\" (UID: \"9fcdb3a5-74bc-44be-993f-fbc90e5db202\") " pod="openshift-multus/multus-additional-cni-plugins-qbp2g"
Jul 30 17:00:27.994443 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.993435    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a2093549-0eca-49bd-96d3-2e0c01a4c19c-host-var-lib-cni-multus\") pod \"multus-nfpq4\" (UID: \"a2093549-0eca-49bd-96d3-2e0c01a4c19c\") " pod="openshift-multus/multus-nfpq4"
Jul 30 17:00:27.994443 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:27.993472    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51afa6bb-724f-47ab-b47b-36dc8a61ca25-original-pull-secret podName:51afa6bb-724f-47ab-b47b-36dc8a61ca25 nodeName:}" failed. No retries permitted until 2026-07-30 17:00:28.493434715 +0000 UTC m=+2.980802097 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/51afa6bb-724f-47ab-b47b-36dc8a61ca25-original-pull-secret") pod "global-pull-secret-syncer-9hlj5" (UID: "51afa6bb-724f-47ab-b47b-36dc8a61ca25") : object "kube-system"/"original-pull-secret" not registered
Jul 30 17:00:27.994443 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.993508    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e1dcce1f-b59e-47cc-9015-429f8a7858cd-host-cni-bin\") pod \"ovnkube-node-t87pw\" (UID: \"e1dcce1f-b59e-47cc-9015-429f8a7858cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-t87pw"
Jul 30 17:00:27.994443 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.993534    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a28bc718-4b4f-408b-b1ca-57f116295c99-registration-dir\") pod \"aws-ebs-csi-driver-node-9hmcr\" (UID: \"a28bc718-4b4f-408b-b1ca-57f116295c99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9hmcr"
Jul 30 17:00:27.994443 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.993557    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/80fefbe4-044f-4c09-b4ae-4bcc3d5684d6-lib-modules\") pod \"tuned-bncwx\" (UID: \"80fefbe4-044f-4c09-b4ae-4bcc3d5684d6\") " pod="openshift-cluster-node-tuning-operator/tuned-bncwx"
Jul 30 17:00:27.994443 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.993582    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a28bc718-4b4f-408b-b1ca-57f116295c99-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9hmcr\" (UID: \"a28bc718-4b4f-408b-b1ca-57f116295c99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9hmcr"
Jul 30 17:00:27.994443 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.993605    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2093549-0eca-49bd-96d3-2e0c01a4c19c-multus-cni-dir\") pod \"multus-nfpq4\" (UID: \"a2093549-0eca-49bd-96d3-2e0c01a4c19c\") " pod="openshift-multus/multus-nfpq4"
Jul 30 17:00:27.994443 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.993629    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a2093549-0eca-49bd-96d3-2e0c01a4c19c-host-run-multus-certs\") pod \"multus-nfpq4\" (UID: \"a2093549-0eca-49bd-96d3-2e0c01a4c19c\") " pod="openshift-multus/multus-nfpq4"
Jul 30 17:00:27.994443 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.993654    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/80fefbe4-044f-4c09-b4ae-4bcc3d5684d6-etc-modprobe-d\") pod \"tuned-bncwx\" (UID: \"80fefbe4-044f-4c09-b4ae-4bcc3d5684d6\") " pod="openshift-cluster-node-tuning-operator/tuned-bncwx"
Jul 30 17:00:27.994443 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.993704    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/80fefbe4-044f-4c09-b4ae-4bcc3d5684d6-lib-modules\") pod \"tuned-bncwx\" (UID: \"80fefbe4-044f-4c09-b4ae-4bcc3d5684d6\") " pod="openshift-cluster-node-tuning-operator/tuned-bncwx"
Jul 30 17:00:27.994443 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.993717    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/80fefbe4-044f-4c09-b4ae-4bcc3d5684d6-etc-sysctl-conf\") pod \"tuned-bncwx\" (UID: \"80fefbe4-044f-4c09-b4ae-4bcc3d5684d6\") " pod="openshift-cluster-node-tuning-operator/tuned-bncwx"
Jul 30 17:00:27.994443 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.993746    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/80fefbe4-044f-4c09-b4ae-4bcc3d5684d6-run\") pod \"tuned-bncwx\" (UID: \"80fefbe4-044f-4c09-b4ae-4bcc3d5684d6\") " pod="openshift-cluster-node-tuning-operator/tuned-bncwx"
Jul 30 17:00:27.994443 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.993755    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/80fefbe4-044f-4c09-b4ae-4bcc3d5684d6-etc-modprobe-d\") pod \"tuned-bncwx\" (UID: \"80fefbe4-044f-4c09-b4ae-4bcc3d5684d6\") " pod="openshift-cluster-node-tuning-operator/tuned-bncwx"
Jul 30 17:00:27.994443 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.993771    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/80fefbe4-044f-4c09-b4ae-4bcc3d5684d6-etc-tuned\") pod \"tuned-bncwx\" (UID: \"80fefbe4-044f-4c09-b4ae-4bcc3d5684d6\") " pod="openshift-cluster-node-tuning-operator/tuned-bncwx"
Jul 30 17:00:27.994443 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.993817    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/80fefbe4-044f-4c09-b4ae-4bcc3d5684d6-run\") pod \"tuned-bncwx\" (UID: \"80fefbe4-044f-4c09-b4ae-4bcc3d5684d6\") " pod="openshift-cluster-node-tuning-operator/tuned-bncwx"
Jul 30 17:00:27.995294 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.993837    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1dcce1f-b59e-47cc-9015-429f8a7858cd-run-openvswitch\") pod \"ovnkube-node-t87pw\" (UID: \"e1dcce1f-b59e-47cc-9015-429f8a7858cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-t87pw"
Jul 30 17:00:27.995294 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.993882    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e1dcce1f-b59e-47cc-9015-429f8a7858cd-host-cni-netd\") pod \"ovnkube-node-t87pw\" (UID: \"e1dcce1f-b59e-47cc-9015-429f8a7858cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-t87pw"
Jul 30 17:00:27.995294 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.993903    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/80fefbe4-044f-4c09-b4ae-4bcc3d5684d6-etc-sysctl-conf\") pod \"tuned-bncwx\" (UID: \"80fefbe4-044f-4c09-b4ae-4bcc3d5684d6\") " pod="openshift-cluster-node-tuning-operator/tuned-bncwx"
Jul 30 17:00:27.995294 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.993916    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e1dcce1f-b59e-47cc-9015-429f8a7858cd-host-slash\") pod \"ovnkube-node-t87pw\" (UID: \"e1dcce1f-b59e-47cc-9015-429f8a7858cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-t87pw"
Jul 30 17:00:27.995294 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.993944    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1dcce1f-b59e-47cc-9015-429f8a7858cd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-t87pw\" (UID: \"e1dcce1f-b59e-47cc-9015-429f8a7858cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-t87pw"
Jul 30 17:00:27.995294 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.993971    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59lv4\" (UniqueName: \"kubernetes.io/projected/ed0f2ced-c500-4288-ac35-3e5599c9d25f-kube-api-access-59lv4\") pod \"iptables-alerter-2tdqf\" (UID: \"ed0f2ced-c500-4288-ac35-3e5599c9d25f\") " pod="openshift-network-operator/iptables-alerter-2tdqf"
Jul 30 17:00:27.995294 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.994014    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9485af8e-c7d9-4b8a-ab94-aa3b30415381-agent-certs\") pod \"konnectivity-agent-8lckj\" (UID: \"9485af8e-c7d9-4b8a-ab94-aa3b30415381\") " pod="kube-system/konnectivity-agent-8lckj"
Jul 30 17:00:27.995294 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.994055    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e85f9d71-fe3d-49d8-93e2-e4ac101603c7-metrics-certs\") pod \"network-metrics-daemon-55hfh\" (UID: \"e85f9d71-fe3d-49d8-93e2-e4ac101603c7\") " pod="openshift-multus/network-metrics-daemon-55hfh"
Jul 30 17:00:27.995294 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.994148    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/80fefbe4-044f-4c09-b4ae-4bcc3d5684d6-var-lib-kubelet\") pod \"tuned-bncwx\" (UID: \"80fefbe4-044f-4c09-b4ae-4bcc3d5684d6\") " pod="openshift-cluster-node-tuning-operator/tuned-bncwx"
Jul 30 17:00:27.995294 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:27.994184    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 17:00:27.995294 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.994185    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a28bc718-4b4f-408b-b1ca-57f116295c99-etc-selinux\") pod \"aws-ebs-csi-driver-node-9hmcr\" (UID: \"a28bc718-4b4f-408b-b1ca-57f116295c99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9hmcr"
Jul 30 17:00:27.995294 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.994229    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/80fefbe4-044f-4c09-b4ae-4bcc3d5684d6-var-lib-kubelet\") pod \"tuned-bncwx\" (UID: \"80fefbe4-044f-4c09-b4ae-4bcc3d5684d6\") " pod="openshift-cluster-node-tuning-operator/tuned-bncwx"
Jul 30 17:00:27.995294 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:27.994241    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e85f9d71-fe3d-49d8-93e2-e4ac101603c7-metrics-certs podName:e85f9d71-fe3d-49d8-93e2-e4ac101603c7 nodeName:}" failed. No retries permitted until 2026-07-30 17:00:28.494222593 +0000 UTC m=+2.981589973 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e85f9d71-fe3d-49d8-93e2-e4ac101603c7-metrics-certs") pod "network-metrics-daemon-55hfh" (UID: "e85f9d71-fe3d-49d8-93e2-e4ac101603c7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 17:00:27.995294 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.994278    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a28bc718-4b4f-408b-b1ca-57f116295c99-sys-fs\") pod \"aws-ebs-csi-driver-node-9hmcr\" (UID: \"a28bc718-4b4f-408b-b1ca-57f116295c99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9hmcr"
Jul 30 17:00:27.995294 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.994305    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e1dcce1f-b59e-47cc-9015-429f8a7858cd-ovn-node-metrics-cert\") pod \"ovnkube-node-t87pw\" (UID: \"e1dcce1f-b59e-47cc-9015-429f8a7858cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-t87pw"
Jul 30 17:00:27.995294 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.994334    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a2093549-0eca-49bd-96d3-2e0c01a4c19c-os-release\") pod \"multus-nfpq4\" (UID: \"a2093549-0eca-49bd-96d3-2e0c01a4c19c\") " pod="openshift-multus/multus-nfpq4"
Jul 30 17:00:27.995924 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.994358    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a2093549-0eca-49bd-96d3-2e0c01a4c19c-cni-binary-copy\") pod \"multus-nfpq4\" (UID: \"a2093549-0eca-49bd-96d3-2e0c01a4c19c\") " pod="openshift-multus/multus-nfpq4"
Jul 30 17:00:27.995924 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.994385    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/80fefbe4-044f-4c09-b4ae-4bcc3d5684d6-etc-systemd\") pod \"tuned-bncwx\" (UID: \"80fefbe4-044f-4c09-b4ae-4bcc3d5684d6\") " pod="openshift-cluster-node-tuning-operator/tuned-bncwx"
Jul 30 17:00:27.995924 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.994530    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/80fefbe4-044f-4c09-b4ae-4bcc3d5684d6-etc-systemd\") pod \"tuned-bncwx\" (UID: \"80fefbe4-044f-4c09-b4ae-4bcc3d5684d6\") " pod="openshift-cluster-node-tuning-operator/tuned-bncwx"
Jul 30 17:00:27.996424 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.996406    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/80fefbe4-044f-4c09-b4ae-4bcc3d5684d6-tmp\") pod \"tuned-bncwx\" (UID: \"80fefbe4-044f-4c09-b4ae-4bcc3d5684d6\") " pod="openshift-cluster-node-tuning-operator/tuned-bncwx"
Jul 30 17:00:27.996518 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:27.996486    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/80fefbe4-044f-4c09-b4ae-4bcc3d5684d6-etc-tuned\") pod \"tuned-bncwx\" (UID: \"80fefbe4-044f-4c09-b4ae-4bcc3d5684d6\") " pod="openshift-cluster-node-tuning-operator/tuned-bncwx"
Jul 30 17:00:28.001037 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:28.001015    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 30 17:00:28.001037 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:28.001040    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 30 17:00:28.001037 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:28.001058    2555 projected.go:196] Error preparing data for projected volume kube-api-access-rmqln for pod openshift-network-diagnostics/network-check-target-qvgrv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 17:00:28.001282 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:28.001144    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fc0a3112-260e-47c8-87fd-bf3666ea17d0-kube-api-access-rmqln podName:fc0a3112-260e-47c8-87fd-bf3666ea17d0 nodeName:}" failed. No retries permitted until 2026-07-30 17:00:28.501124364 +0000 UTC m=+2.988491766 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-rmqln" (UniqueName: "kubernetes.io/projected/fc0a3112-260e-47c8-87fd-bf3666ea17d0-kube-api-access-rmqln") pod "network-check-target-qvgrv" (UID: "fc0a3112-260e-47c8-87fd-bf3666ea17d0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 17:00:28.003703 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.003676    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29sfv\" (UniqueName: \"kubernetes.io/projected/80fefbe4-044f-4c09-b4ae-4bcc3d5684d6-kube-api-access-29sfv\") pod \"tuned-bncwx\" (UID: \"80fefbe4-044f-4c09-b4ae-4bcc3d5684d6\") " pod="openshift-cluster-node-tuning-operator/tuned-bncwx"
Jul 30 17:00:28.003819 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.003799    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrkrk\" (UniqueName: \"kubernetes.io/projected/e85f9d71-fe3d-49d8-93e2-e4ac101603c7-kube-api-access-jrkrk\") pod \"network-metrics-daemon-55hfh\" (UID: \"e85f9d71-fe3d-49d8-93e2-e4ac101603c7\") " pod="openshift-multus/network-metrics-daemon-55hfh"
Jul 30 17:00:28.060345 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.060273    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-190.ec2.internal" event={"ID":"041d73f2cda3ec42dcee6249e2df6511","Type":"ContainerStarted","Data":"e386cd85406b710c928e3d4e0918530bf107137ddfbe27213c922c9a6f39cdcb"}
Jul 30 17:00:28.061674 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.061639    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-190.ec2.internal" event={"ID":"3e02b1d2169088ae669788950ba2bc8f","Type":"ContainerStarted","Data":"b36841d9d138eca2dab1a9e62a6a043a581122afb4fb0c722026052f18a1ef89"}
Jul 30 17:00:28.095332 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.095294    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e1dcce1f-b59e-47cc-9015-429f8a7858cd-systemd-units\") pod \"ovnkube-node-t87pw\" (UID: \"e1dcce1f-b59e-47cc-9015-429f8a7858cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-t87pw"
Jul 30 17:00:28.095531 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.095342    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1dcce1f-b59e-47cc-9015-429f8a7858cd-var-lib-openvswitch\") pod \"ovnkube-node-t87pw\" (UID: \"e1dcce1f-b59e-47cc-9015-429f8a7858cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-t87pw"
Jul 30 17:00:28.095531 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.095369    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a2093549-0eca-49bd-96d3-2e0c01a4c19c-host-run-k8s-cni-cncf-io\") pod \"multus-nfpq4\" (UID: \"a2093549-0eca-49bd-96d3-2e0c01a4c19c\") " pod="openshift-multus/multus-nfpq4"
Jul 30 17:00:28.095531 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.095383    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e1dcce1f-b59e-47cc-9015-429f8a7858cd-systemd-units\") pod \"ovnkube-node-t87pw\" (UID: \"e1dcce1f-b59e-47cc-9015-429f8a7858cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-t87pw"
Jul 30 17:00:28.095531 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.095396    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a28bc718-4b4f-408b-b1ca-57f116295c99-device-dir\") pod \"aws-ebs-csi-driver-node-9hmcr\" (UID: \"a28bc718-4b4f-408b-b1ca-57f116295c99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9hmcr"
Jul 30 17:00:28.095531 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.095444    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1dcce1f-b59e-47cc-9015-429f8a7858cd-var-lib-openvswitch\") pod \"ovnkube-node-t87pw\" (UID: \"e1dcce1f-b59e-47cc-9015-429f8a7858cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-t87pw"
Jul 30 17:00:28.095531 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.095458    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1dcce1f-b59e-47cc-9015-429f8a7858cd-host-run-ovn-kubernetes\") pod \"ovnkube-node-t87pw\" (UID: \"e1dcce1f-b59e-47cc-9015-429f8a7858cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-t87pw"
Jul 30 17:00:28.095531 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.095491    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ed0f2ced-c500-4288-ac35-3e5599c9d25f-iptables-alerter-script\") pod \"iptables-alerter-2tdqf\" (UID: \"ed0f2ced-c500-4288-ac35-3e5599c9d25f\") " pod="openshift-network-operator/iptables-alerter-2tdqf"
Jul 30 17:00:28.095531 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.095505    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a28bc718-4b4f-408b-b1ca-57f116295c99-device-dir\") pod \"aws-ebs-csi-driver-node-9hmcr\" (UID: \"a28bc718-4b4f-408b-b1ca-57f116295c99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9hmcr"
Jul 30 17:00:28.095531 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.095518    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9fcdb3a5-74bc-44be-993f-fbc90e5db202-os-release\") pod \"multus-additional-cni-plugins-qbp2g\" (UID: \"9fcdb3a5-74bc-44be-993f-fbc90e5db202\") " pod="openshift-multus/multus-additional-cni-plugins-qbp2g"
Jul 30 17:00:28.095926 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.095545    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2093549-0eca-49bd-96d3-2e0c01a4c19c-system-cni-dir\") pod \"multus-nfpq4\" (UID: \"a2093549-0eca-49bd-96d3-2e0c01a4c19c\") " pod="openshift-multus/multus-nfpq4"
Jul 30 17:00:28.095926 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.095572    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a2093549-0eca-49bd-96d3-2e0c01a4c19c-host-var-lib-cni-bin\") pod \"multus-nfpq4\" (UID: \"a2093549-0eca-49bd-96d3-2e0c01a4c19c\") " pod="openshift-multus/multus-nfpq4"
Jul 30 17:00:28.095926 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.095597    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2093549-0eca-49bd-96d3-2e0c01a4c19c-host-var-lib-kubelet\") pod \"multus-nfpq4\" (UID: \"a2093549-0eca-49bd-96d3-2e0c01a4c19c\") " pod="openshift-multus/multus-nfpq4"
Jul 30 17:00:28.095926 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.095625    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e1dcce1f-b59e-47cc-9015-429f8a7858cd-ovnkube-config\") pod \"ovnkube-node-t87pw\" (UID: \"e1dcce1f-b59e-47cc-9015-429f8a7858cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-t87pw"
Jul 30 17:00:28.095926 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.095649    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e1dcce1f-b59e-47cc-9015-429f8a7858cd-env-overrides\") pod \"ovnkube-node-t87pw\" (UID: \"e1dcce1f-b59e-47cc-9015-429f8a7858cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-t87pw"
Jul 30 17:00:28.095926 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.095673    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bzxp\" (UniqueName: \"kubernetes.io/projected/e1dcce1f-b59e-47cc-9015-429f8a7858cd-kube-api-access-6bzxp\") pod \"ovnkube-node-t87pw\" (UID: \"e1dcce1f-b59e-47cc-9015-429f8a7858cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-t87pw"
Jul 30 17:00:28.095926 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.095705    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9fcdb3a5-74bc-44be-993f-fbc90e5db202-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qbp2g\" (UID: \"9fcdb3a5-74bc-44be-993f-fbc90e5db202\") " pod="openshift-multus/multus-additional-cni-plugins-qbp2g"
Jul 30 17:00:28.095926 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.095733    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a2093549-0eca-49bd-96d3-2e0c01a4c19c-host-run-netns\") pod \"multus-nfpq4\" (UID: \"a2093549-0eca-49bd-96d3-2e0c01a4c19c\") " pod="openshift-multus/multus-nfpq4"
Jul 30 17:00:28.095926 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.095758    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd6ws\" (UniqueName: \"kubernetes.io/projected/a2093549-0eca-49bd-96d3-2e0c01a4c19c-kube-api-access-nd6ws\") pod \"multus-nfpq4\" (UID: \"a2093549-0eca-49bd-96d3-2e0c01a4c19c\") " pod="openshift-multus/multus-nfpq4"
Jul 30 17:00:28.095926 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.095783    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ed0f2ced-c500-4288-ac35-3e5599c9d25f-host-slash\") pod \"iptables-alerter-2tdqf\" (UID: \"ed0f2ced-c500-4288-ac35-3e5599c9d25f\") " pod="openshift-network-operator/iptables-alerter-2tdqf"
Jul 30 17:00:28.095926 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.095812    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/97f5f5b8-ce32-41cd-9b5c-c94f2ae6bbfe-host\") pod \"node-ca-xglts\" (UID: \"97f5f5b8-ce32-41cd-9b5c-c94f2ae6bbfe\") " pod="openshift-image-registry/node-ca-xglts"
Jul 30 17:00:28.095926 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.095838    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv555\" (UniqueName: \"kubernetes.io/projected/97f5f5b8-ce32-41cd-9b5c-c94f2ae6bbfe-kube-api-access-vv555\") pod \"node-ca-xglts\" (UID: \"97f5f5b8-ce32-41cd-9b5c-c94f2ae6bbfe\") " pod="openshift-image-registry/node-ca-xglts"
Jul 30 17:00:28.095926 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.095864    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhhj5\" (UniqueName: \"kubernetes.io/projected/9fcdb3a5-74bc-44be-993f-fbc90e5db202-kube-api-access-mhhj5\") pod \"multus-additional-cni-plugins-qbp2g\" (UID: \"9fcdb3a5-74bc-44be-993f-fbc90e5db202\") " pod="openshift-multus/multus-additional-cni-plugins-qbp2g"
Jul 30 17:00:28.095926 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.095891    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a2093549-0eca-49bd-96d3-2e0c01a4c19c-cnibin\") pod \"multus-nfpq4\" (UID: \"a2093549-0eca-49bd-96d3-2e0c01a4c19c\") " pod="openshift-multus/multus-nfpq4"
Jul 30 17:00:28.095926 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.095925    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a2093549-0eca-49bd-96d3-2e0c01a4c19c-multus-socket-dir-parent\") pod \"multus-nfpq4\" (UID: \"a2093549-0eca-49bd-96d3-2e0c01a4c19c\") " pod="openshift-multus/multus-nfpq4"
Jul 30 17:00:28.096629 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.095954    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a2093549-0eca-49bd-96d3-2e0c01a4c19c-multus-conf-dir\") pod \"multus-nfpq4\" (UID: \"a2093549-0eca-49bd-96d3-2e0c01a4c19c\") " pod="openshift-multus/multus-nfpq4"
Jul 30 17:00:28.096629 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.095978    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1dcce1f-b59e-47cc-9015-429f8a7858cd-host-kubelet\") pod \"ovnkube-node-t87pw\" (UID: \"e1dcce1f-b59e-47cc-9015-429f8a7858cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-t87pw"
Jul 30 17:00:28.096629 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.096002    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e1dcce1f-b59e-47cc-9015-429f8a7858cd-run-systemd\") pod \"ovnkube-node-t87pw\" (UID: \"e1dcce1f-b59e-47cc-9015-429f8a7858cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-t87pw"
Jul 30 17:00:28.096629 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.096026    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3534e135-e217-49b8-8866-8cd33104b77e-tmp-dir\") pod \"node-resolver-97k9j\" (UID: \"3534e135-e217-49b8-8866-8cd33104b77e\") " pod="openshift-dns/node-resolver-97k9j"
Jul 30 17:00:28.096629 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.096052    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9fcdb3a5-74bc-44be-993f-fbc90e5db202-system-cni-dir\") pod \"multus-additional-cni-plugins-qbp2g\" (UID: \"9fcdb3a5-74bc-44be-993f-fbc90e5db202\") " pod="openshift-multus/multus-additional-cni-plugins-qbp2g"
Jul 30 17:00:28.096629 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.096128    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9485af8e-c7d9-4b8a-ab94-aa3b30415381-konnectivity-ca\") pod \"konnectivity-agent-8lckj\" (UID: \"9485af8e-c7d9-4b8a-ab94-aa3b30415381\") " pod="kube-system/konnectivity-agent-8lckj"
Jul 30 17:00:28.096629 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.096144    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ed0f2ced-c500-4288-ac35-3e5599c9d25f-iptables-alerter-script\") pod \"iptables-alerter-2tdqf\" (UID: \"ed0f2ced-c500-4288-ac35-3e5599c9d25f\") " pod="openshift-network-operator/iptables-alerter-2tdqf"
Jul 30 17:00:28.096629 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.096155    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9fcdb3a5-74bc-44be-993f-fbc90e5db202-cnibin\") pod \"multus-additional-cni-plugins-qbp2g\" (UID: \"9fcdb3a5-74bc-44be-993f-fbc90e5db202\") " pod="openshift-multus/multus-additional-cni-plugins-qbp2g"
Jul 30 17:00:28.096629 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.096194    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9fcdb3a5-74bc-44be-993f-fbc90e5db202-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qbp2g\" (UID: \"9fcdb3a5-74bc-44be-993f-fbc90e5db202\") " pod="openshift-multus/multus-additional-cni-plugins-qbp2g"
Jul 30 17:00:28.096629 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.096225    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9fcdb3a5-74bc-44be-993f-fbc90e5db202-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qbp2g\" (UID: \"9fcdb3a5-74bc-44be-993f-fbc90e5db202\") " pod="openshift-multus/multus-additional-cni-plugins-qbp2g"
Jul 30 17:00:28.096629 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.096241    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1dcce1f-b59e-47cc-9015-429f8a7858cd-host-run-ovn-kubernetes\") pod \"ovnkube-node-t87pw\" (UID: \"e1dcce1f-b59e-47cc-9015-429f8a7858cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-t87pw"
Jul 30 17:00:28.096629 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.096252    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a2093549-0eca-49bd-96d3-2e0c01a4c19c-host-var-lib-cni-multus\") pod \"multus-nfpq4\" (UID: \"a2093549-0eca-49bd-96d3-2e0c01a4c19c\") " pod="openshift-multus/multus-nfpq4"
Jul 30 17:00:28.096629 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.096281    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e1dcce1f-b59e-47cc-9015-429f8a7858cd-host-cni-bin\") pod \"ovnkube-node-t87pw\" (UID: \"e1dcce1f-b59e-47cc-9015-429f8a7858cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-t87pw"
Jul 30 17:00:28.096629 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.096295    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/97f5f5b8-ce32-41cd-9b5c-c94f2ae6bbfe-host\") pod \"node-ca-xglts\" (UID: \"97f5f5b8-ce32-41cd-9b5c-c94f2ae6bbfe\") " pod="openshift-image-registry/node-ca-xglts"
Jul 30 17:00:28.096629 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.096309    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a28bc718-4b4f-408b-b1ca-57f116295c99-registration-dir\") pod \"aws-ebs-csi-driver-node-9hmcr\" (UID: \"a28bc718-4b4f-408b-b1ca-57f116295c99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9hmcr"
Jul 30 17:00:28.096629 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.096338    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a28bc718-4b4f-408b-b1ca-57f116295c99-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9hmcr\" (UID: \"a28bc718-4b4f-408b-b1ca-57f116295c99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9hmcr"
Jul 30 17:00:28.096629 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.096362    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9fcdb3a5-74bc-44be-993f-fbc90e5db202-os-release\") pod \"multus-additional-cni-plugins-qbp2g\" (UID: \"9fcdb3a5-74bc-44be-993f-fbc90e5db202\") " pod="openshift-multus/multus-additional-cni-plugins-qbp2g"
Jul 30 17:00:28.097488 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.096363    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2093549-0eca-49bd-96d3-2e0c01a4c19c-multus-cni-dir\") pod \"multus-nfpq4\" (UID: \"a2093549-0eca-49bd-96d3-2e0c01a4c19c\") " pod="openshift-multus/multus-nfpq4"
Jul 30 17:00:28.097488 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.096407    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a2093549-0eca-49bd-96d3-2e0c01a4c19c-host-run-multus-certs\") pod \"multus-nfpq4\" (UID: \"a2093549-0eca-49bd-96d3-2e0c01a4c19c\") " pod="openshift-multus/multus-nfpq4"
Jul 30 17:00:28.097488 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.096432    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2093549-0eca-49bd-96d3-2e0c01a4c19c-multus-cni-dir\") pod \"multus-nfpq4\" (UID: \"a2093549-0eca-49bd-96d3-2e0c01a4c19c\") " pod="openshift-multus/multus-nfpq4"
Jul 30 17:00:28.097488 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.096446    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1dcce1f-b59e-47cc-9015-429f8a7858cd-run-openvswitch\") pod \"ovnkube-node-t87pw\" (UID: \"e1dcce1f-b59e-47cc-9015-429f8a7858cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-t87pw"
Jul 30 17:00:28.097488 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.096475    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e1dcce1f-b59e-47cc-9015-429f8a7858cd-host-cni-netd\") pod \"ovnkube-node-t87pw\" (UID: \"e1dcce1f-b59e-47cc-9015-429f8a7858cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-t87pw"
Jul 30 17:00:28.097488 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.096505    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e1dcce1f-b59e-47cc-9015-429f8a7858cd-host-slash\") pod \"ovnkube-node-t87pw\" (UID: \"e1dcce1f-b59e-47cc-9015-429f8a7858cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-t87pw"
Jul 30 17:00:28.097488 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.096535    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1dcce1f-b59e-47cc-9015-429f8a7858cd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-t87pw\" (UID: \"e1dcce1f-b59e-47cc-9015-429f8a7858cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-t87pw"
Jul 30 17:00:28.097488 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.096578    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59lv4\" (UniqueName: \"kubernetes.io/projected/ed0f2ced-c500-4288-ac35-3e5599c9d25f-kube-api-access-59lv4\") pod \"iptables-alerter-2tdqf\" (UID: \"ed0f2ced-c500-4288-ac35-3e5599c9d25f\") " pod="openshift-network-operator/iptables-alerter-2tdqf"
Jul 30 17:00:28.097488 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.096610    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9485af8e-c7d9-4b8a-ab94-aa3b30415381-agent-certs\") pod \"konnectivity-agent-8lckj\" (UID: \"9485af8e-c7d9-4b8a-ab94-aa3b30415381\") " pod="kube-system/konnectivity-agent-8lckj"
Jul 30 17:00:28.097488 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.096635    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a2093549-0eca-49bd-96d3-2e0c01a4c19c-host-run-netns\") pod \"multus-nfpq4\" (UID: \"a2093549-0eca-49bd-96d3-2e0c01a4c19c\") " pod="openshift-multus/multus-nfpq4"
Jul 30 17:00:28.097488 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.096652    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a28bc718-4b4f-408b-b1ca-57f116295c99-etc-selinux\") pod \"aws-ebs-csi-driver-node-9hmcr\" (UID: \"a28bc718-4b4f-408b-b1ca-57f116295c99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9hmcr"
Jul 30 17:00:28.097488 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.096679    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a28bc718-4b4f-408b-b1ca-57f116295c99-sys-fs\") pod \"aws-ebs-csi-driver-node-9hmcr\" (UID: \"a28bc718-4b4f-408b-b1ca-57f116295c99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9hmcr"
Jul 30 17:00:28.097488 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.096703    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2093549-0eca-49bd-96d3-2e0c01a4c19c-system-cni-dir\") pod \"multus-nfpq4\" (UID: \"a2093549-0eca-49bd-96d3-2e0c01a4c19c\") " pod="openshift-multus/multus-nfpq4"
Jul 30 17:00:28.097488 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.096705    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e1dcce1f-b59e-47cc-9015-429f8a7858cd-ovn-node-metrics-cert\") pod \"ovnkube-node-t87pw\" (UID: \"e1dcce1f-b59e-47cc-9015-429f8a7858cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-t87pw"
Jul 30 17:00:28.097488 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.096741    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a2093549-0eca-49bd-96d3-2e0c01a4c19c-os-release\") pod \"multus-nfpq4\" (UID: \"a2093549-0eca-49bd-96d3-2e0c01a4c19c\") " pod="openshift-multus/multus-nfpq4"
Jul 30 17:00:28.097488 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.096762    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a2093549-0eca-49bd-96d3-2e0c01a4c19c-cni-binary-copy\") pod \"multus-nfpq4\" (UID: \"a2093549-0eca-49bd-96d3-2e0c01a4c19c\") " pod="openshift-multus/multus-nfpq4"
Jul 30 17:00:28.097488 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.096773    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a2093549-0eca-49bd-96d3-2e0c01a4c19c-cnibin\") pod \"multus-nfpq4\" (UID: \"a2093549-0eca-49bd-96d3-2e0c01a4c19c\") " pod="openshift-multus/multus-nfpq4"
Jul 30 17:00:28.097488 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.096808    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2093549-0eca-49bd-96d3-2e0c01a4c19c-etc-kubernetes\") pod \"multus-nfpq4\" (UID: \"a2093549-0eca-49bd-96d3-2e0c01a4c19c\") " pod="openshift-multus/multus-nfpq4"
Jul 30 17:00:28.098413 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.096831    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfkrg\" (UniqueName: \"kubernetes.io/projected/a28bc718-4b4f-408b-b1ca-57f116295c99-kube-api-access-mfkrg\") pod \"aws-ebs-csi-driver-node-9hmcr\" (UID: \"a28bc718-4b4f-408b-b1ca-57f116295c99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9hmcr"
Jul 30 17:00:28.098413 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.096857    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1dcce1f-b59e-47cc-9015-429f8a7858cd-etc-openvswitch\") pod \"ovnkube-node-t87pw\" (UID: \"e1dcce1f-b59e-47cc-9015-429f8a7858cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-t87pw"
Jul 30 17:00:28.098413 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.096882    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e1dcce1f-b59e-47cc-9015-429f8a7858cd-node-log\") pod \"ovnkube-node-t87pw\" (UID: \"e1dcce1f-b59e-47cc-9015-429f8a7858cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-t87pw"
Jul 30 17:00:28.098413 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.096901    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/97f5f5b8-ce32-41cd-9b5c-c94f2ae6bbfe-serviceca\") pod \"node-ca-xglts\" (UID: \"97f5f5b8-ce32-41cd-9b5c-c94f2ae6bbfe\") " pod="openshift-image-registry/node-ca-xglts"
Jul 30 17:00:28.098413 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.096917    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a2093549-0eca-49bd-96d3-2e0c01a4c19c-hostroot\") pod \"multus-nfpq4\" (UID: \"a2093549-0eca-49bd-96d3-2e0c01a4c19c\") " pod="openshift-multus/multus-nfpq4"
Jul 30 17:00:28.098413 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.096935    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a2093549-0eca-49bd-96d3-2e0c01a4c19c-multus-daemon-config\") pod \"multus-nfpq4\" (UID: \"a2093549-0eca-49bd-96d3-2e0c01a4c19c\") " pod="openshift-multus/multus-nfpq4"
Jul 30 17:00:28.098413 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.096957    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3534e135-e217-49b8-8866-8cd33104b77e-hosts-file\") pod \"node-resolver-97k9j\" (UID: \"3534e135-e217-49b8-8866-8cd33104b77e\") " pod="openshift-dns/node-resolver-97k9j"
Jul 30 17:00:28.098413 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.095548    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a2093549-0eca-49bd-96d3-2e0c01a4c19c-host-run-k8s-cni-cncf-io\") pod \"multus-nfpq4\" (UID: \"a2093549-0eca-49bd-96d3-2e0c01a4c19c\") " pod="openshift-multus/multus-nfpq4"
Jul 30 17:00:28.098413 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.097000    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a2093549-0eca-49bd-96d3-2e0c01a4c19c-host-var-lib-cni-bin\") pod \"multus-nfpq4\" (UID: \"a2093549-0eca-49bd-96d3-2e0c01a4c19c\") " pod="openshift-multus/multus-nfpq4"
Jul 30 17:00:28.098413 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.097002    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ed0f2ced-c500-4288-ac35-3e5599c9d25f-host-slash\") pod \"iptables-alerter-2tdqf\" (UID: \"ed0f2ced-c500-4288-ac35-3e5599c9d25f\") " pod="openshift-network-operator/iptables-alerter-2tdqf"
Jul 30 17:00:28.098413 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.097025    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2093549-0eca-49bd-96d3-2e0c01a4c19c-host-var-lib-kubelet\") pod \"multus-nfpq4\" (UID: \"a2093549-0eca-49bd-96d3-2e0c01a4c19c\") " pod="openshift-multus/multus-nfpq4"
Jul 30 17:00:28.098413 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.097083    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3534e135-e217-49b8-8866-8cd33104b77e-hosts-file\") pod \"node-resolver-97k9j\" (UID: \"3534e135-e217-49b8-8866-8cd33104b77e\") " pod="openshift-dns/node-resolver-97k9j"
Jul 30 17:00:28.098413 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.097127    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a2093549-0eca-49bd-96d3-2e0c01a4c19c-host-run-multus-certs\") pod \"multus-nfpq4\" (UID: \"a2093549-0eca-49bd-96d3-2e0c01a4c19c\") " pod="openshift-multus/multus-nfpq4"
Jul 30 17:00:28.098413 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.097170    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1dcce1f-b59e-47cc-9015-429f8a7858cd-run-openvswitch\") pod \"ovnkube-node-t87pw\" (UID: \"e1dcce1f-b59e-47cc-9015-429f8a7858cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-t87pw"
Jul 30 17:00:28.098413 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.097257    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2093549-0eca-49bd-96d3-2e0c01a4c19c-etc-kubernetes\") pod \"multus-nfpq4\" (UID: \"a2093549-0eca-49bd-96d3-2e0c01a4c19c\") " pod="openshift-multus/multus-nfpq4"
Jul 30 17:00:28.098413 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.097588    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e1dcce1f-b59e-47cc-9015-429f8a7858cd-ovnkube-config\") pod \"ovnkube-node-t87pw\" (UID: \"e1dcce1f-b59e-47cc-9015-429f8a7858cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-t87pw"
Jul 30 17:00:28.098413 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.097600    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e1dcce1f-b59e-47cc-9015-429f8a7858cd-host-cni-netd\") pod \"ovnkube-node-t87pw\" (UID: \"e1dcce1f-b59e-47cc-9015-429f8a7858cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-t87pw"
Jul 30 17:00:28.098413 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.097680    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e1dcce1f-b59e-47cc-9015-429f8a7858cd-host-slash\") pod \"ovnkube-node-t87pw\" (UID: \"e1dcce1f-b59e-47cc-9015-429f8a7858cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-t87pw"
Jul 30 17:00:28.099370 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.097683    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e1dcce1f-b59e-47cc-9015-429f8a7858cd-env-overrides\") pod \"ovnkube-node-t87pw\" (UID: \"e1dcce1f-b59e-47cc-9015-429f8a7858cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-t87pw"
Jul 30 17:00:28.099370 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.097731    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1dcce1f-b59e-47cc-9015-429f8a7858cd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-t87pw\" (UID: \"e1dcce1f-b59e-47cc-9015-429f8a7858cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-t87pw"
Jul 30 17:00:28.099370 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.097843    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1dcce1f-b59e-47cc-9015-429f8a7858cd-etc-openvswitch\") pod \"ovnkube-node-t87pw\" (UID: \"e1dcce1f-b59e-47cc-9015-429f8a7858cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-t87pw"
Jul 30 17:00:28.099370 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.097893    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e1dcce1f-b59e-47cc-9015-429f8a7858cd-node-log\") pod \"ovnkube-node-t87pw\" (UID: \"e1dcce1f-b59e-47cc-9015-429f8a7858cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-t87pw"
Jul 30 17:00:28.099370 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.098038    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9fcdb3a5-74bc-44be-993f-fbc90e5db202-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qbp2g\" (UID: \"9fcdb3a5-74bc-44be-993f-fbc90e5db202\") " pod="openshift-multus/multus-additional-cni-plugins-qbp2g"
Jul 30 17:00:28.099370 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.098130    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a2093549-0eca-49bd-96d3-2e0c01a4c19c-os-release\") pod \"multus-nfpq4\" (UID: \"a2093549-0eca-49bd-96d3-2e0c01a4c19c\") " pod="openshift-multus/multus-nfpq4"
Jul 30 17:00:28.099370 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.098203    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1dcce1f-b59e-47cc-9015-429f8a7858cd-host-kubelet\") pod \"ovnkube-node-t87pw\" (UID: \"e1dcce1f-b59e-47cc-9015-429f8a7858cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-t87pw"
Jul 30 17:00:28.099370 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.098274    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a2093549-0eca-49bd-96d3-2e0c01a4c19c-multus-socket-dir-parent\") pod \"multus-nfpq4\" (UID: \"a2093549-0eca-49bd-96d3-2e0c01a4c19c\") " pod="openshift-multus/multus-nfpq4"
Jul 30 17:00:28.099370 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.098330    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a2093549-0eca-49bd-96d3-2e0c01a4c19c-multus-conf-dir\") pod \"multus-nfpq4\" (UID: \"a2093549-0eca-49bd-96d3-2e0c01a4c19c\") " pod="openshift-multus/multus-nfpq4"
Jul 30 17:00:28.099370 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.098330    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/97f5f5b8-ce32-41cd-9b5c-c94f2ae6bbfe-serviceca\") pod \"node-ca-xglts\" (UID: \"97f5f5b8-ce32-41cd-9b5c-c94f2ae6bbfe\") " pod="openshift-image-registry/node-ca-xglts"
Jul 30 17:00:28.099370 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.098371    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a2093549-0eca-49bd-96d3-2e0c01a4c19c-hostroot\") pod \"multus-nfpq4\" (UID: \"a2093549-0eca-49bd-96d3-2e0c01a4c19c\") " pod="openshift-multus/multus-nfpq4"
Jul 30 17:00:28.099370 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.098442    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9485af8e-c7d9-4b8a-ab94-aa3b30415381-konnectivity-ca\") pod \"konnectivity-agent-8lckj\" (UID: \"9485af8e-c7d9-4b8a-ab94-aa3b30415381\") " pod="kube-system/konnectivity-agent-8lckj"
Jul 30 17:00:28.099370 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.098515    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9fcdb3a5-74bc-44be-993f-fbc90e5db202-system-cni-dir\") pod \"multus-additional-cni-plugins-qbp2g\" (UID: \"9fcdb3a5-74bc-44be-993f-fbc90e5db202\") " pod="openshift-multus/multus-additional-cni-plugins-qbp2g"
Jul 30 17:00:28.099370 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.098575    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a28bc718-4b4f-408b-b1ca-57f116295c99-sys-fs\") pod \"aws-ebs-csi-driver-node-9hmcr\" (UID: \"a28bc718-4b4f-408b-b1ca-57f116295c99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9hmcr"
Jul 30 17:00:28.099370 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.098591    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgmjc\" (UniqueName: \"kubernetes.io/projected/3534e135-e217-49b8-8866-8cd33104b77e-kube-api-access-kgmjc\") pod \"node-resolver-97k9j\" (UID: \"3534e135-e217-49b8-8866-8cd33104b77e\") " pod="openshift-dns/node-resolver-97k9j"
Jul 30 17:00:28.099370 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.098629    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9fcdb3a5-74bc-44be-993f-fbc90e5db202-cni-binary-copy\") pod \"multus-additional-cni-plugins-qbp2g\" (UID: \"9fcdb3a5-74bc-44be-993f-fbc90e5db202\") " pod="openshift-multus/multus-additional-cni-plugins-qbp2g"
Jul 30 17:00:28.099370 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.098665    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e1dcce1f-b59e-47cc-9015-429f8a7858cd-host-run-netns\") pod \"ovnkube-node-t87pw\" (UID: \"e1dcce1f-b59e-47cc-9015-429f8a7858cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-t87pw"
Jul 30 17:00:28.099370 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.098673    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3534e135-e217-49b8-8866-8cd33104b77e-tmp-dir\") pod \"node-resolver-97k9j\" (UID: \"3534e135-e217-49b8-8866-8cd33104b77e\") " pod="openshift-dns/node-resolver-97k9j"
Jul 30 17:00:28.100289 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.098692    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e1dcce1f-b59e-47cc-9015-429f8a7858cd-run-ovn\") pod \"ovnkube-node-t87pw\" (UID: \"e1dcce1f-b59e-47cc-9015-429f8a7858cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-t87pw"
Jul 30 17:00:28.100289 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.098718    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e1dcce1f-b59e-47cc-9015-429f8a7858cd-log-socket\") pod \"ovnkube-node-t87pw\" (UID: \"e1dcce1f-b59e-47cc-9015-429f8a7858cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-t87pw"
Jul 30 17:00:28.100289 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.098737    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e1dcce1f-b59e-47cc-9015-429f8a7858cd-run-systemd\") pod \"ovnkube-node-t87pw\" (UID: \"e1dcce1f-b59e-47cc-9015-429f8a7858cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-t87pw"
Jul 30 17:00:28.100289 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.098745    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e1dcce1f-b59e-47cc-9015-429f8a7858cd-ovnkube-script-lib\") pod \"ovnkube-node-t87pw\" (UID: \"e1dcce1f-b59e-47cc-9015-429f8a7858cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-t87pw"
Jul 30 17:00:28.100289 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.098775    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a28bc718-4b4f-408b-b1ca-57f116295c99-socket-dir\") pod \"aws-ebs-csi-driver-node-9hmcr\" (UID: \"a28bc718-4b4f-408b-b1ca-57f116295c99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9hmcr"
Jul 30 17:00:28.100289 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.098795    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9fcdb3a5-74bc-44be-993f-fbc90e5db202-cnibin\") pod \"multus-additional-cni-plugins-qbp2g\" (UID: \"9fcdb3a5-74bc-44be-993f-fbc90e5db202\") " pod="openshift-multus/multus-additional-cni-plugins-qbp2g"
Jul 30 17:00:28.100289 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.098863    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a2093549-0eca-49bd-96d3-2e0c01a4c19c-cni-binary-copy\") pod \"multus-nfpq4\" (UID: \"a2093549-0eca-49bd-96d3-2e0c01a4c19c\") " pod="openshift-multus/multus-nfpq4"
Jul 30 17:00:28.100289 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.098894    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a2093549-0eca-49bd-96d3-2e0c01a4c19c-multus-daemon-config\") pod \"multus-nfpq4\" (UID: \"a2093549-0eca-49bd-96d3-2e0c01a4c19c\") " pod="openshift-multus/multus-nfpq4"
Jul 30 17:00:28.100289 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.098939    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a28bc718-4b4f-408b-b1ca-57f116295c99-registration-dir\") pod \"aws-ebs-csi-driver-node-9hmcr\" (UID: \"a28bc718-4b4f-408b-b1ca-57f116295c99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9hmcr"
Jul 30 17:00:28.100289 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.098980    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e1dcce1f-b59e-47cc-9015-429f8a7858cd-host-cni-bin\") pod \"ovnkube-node-t87pw\" (UID: \"e1dcce1f-b59e-47cc-9015-429f8a7858cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-t87pw"
Jul 30 17:00:28.100289 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.099014    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a28bc718-4b4f-408b-b1ca-57f116295c99-etc-selinux\") pod \"aws-ebs-csi-driver-node-9hmcr\" (UID: \"a28bc718-4b4f-408b-b1ca-57f116295c99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9hmcr"
Jul 30 17:00:28.100289 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.099032    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a28bc718-4b4f-408b-b1ca-57f116295c99-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9hmcr\" (UID: \"a28bc718-4b4f-408b-b1ca-57f116295c99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9hmcr"
Jul 30 17:00:28.100289 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.099058    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e1dcce1f-b59e-47cc-9015-429f8a7858cd-run-ovn\") pod \"ovnkube-node-t87pw\" (UID: \"e1dcce1f-b59e-47cc-9015-429f8a7858cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-t87pw"
Jul 30 17:00:28.100289 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.099361    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e1dcce1f-b59e-47cc-9015-429f8a7858cd-log-socket\") pod \"ovnkube-node-t87pw\" (UID: \"e1dcce1f-b59e-47cc-9015-429f8a7858cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-t87pw"
Jul 30 17:00:28.100289 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.099403    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9fcdb3a5-74bc-44be-993f-fbc90e5db202-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qbp2g\" (UID: \"9fcdb3a5-74bc-44be-993f-fbc90e5db202\") " pod="openshift-multus/multus-additional-cni-plugins-qbp2g"
Jul 30 17:00:28.100289 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.099467    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a2093549-0eca-49bd-96d3-2e0c01a4c19c-host-var-lib-cni-multus\") pod \"multus-nfpq4\" (UID: \"a2093549-0eca-49bd-96d3-2e0c01a4c19c\") " pod="openshift-multus/multus-nfpq4"
Jul 30 17:00:28.100289 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.099515    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e1dcce1f-b59e-47cc-9015-429f8a7858cd-host-run-netns\") pod \"ovnkube-node-t87pw\" (UID: \"e1dcce1f-b59e-47cc-9015-429f8a7858cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-t87pw"
Jul 30 17:00:28.101145 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.099633    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a28bc718-4b4f-408b-b1ca-57f116295c99-socket-dir\") pod \"aws-ebs-csi-driver-node-9hmcr\" (UID: \"a28bc718-4b4f-408b-b1ca-57f116295c99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9hmcr"
Jul 30 17:00:28.101145 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.099724    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9fcdb3a5-74bc-44be-993f-fbc90e5db202-cni-binary-copy\") pod \"multus-additional-cni-plugins-qbp2g\" (UID: \"9fcdb3a5-74bc-44be-993f-fbc90e5db202\") " pod="openshift-multus/multus-additional-cni-plugins-qbp2g"
Jul 30 17:00:28.101145 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.099742    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e1dcce1f-b59e-47cc-9015-429f8a7858cd-ovn-node-metrics-cert\") pod \"ovnkube-node-t87pw\" (UID: \"e1dcce1f-b59e-47cc-9015-429f8a7858cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-t87pw"
Jul 30 17:00:28.101145 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.099978    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9fcdb3a5-74bc-44be-993f-fbc90e5db202-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qbp2g\" (UID: \"9fcdb3a5-74bc-44be-993f-fbc90e5db202\") " pod="openshift-multus/multus-additional-cni-plugins-qbp2g"
Jul 30 17:00:28.101145 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.100026    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e1dcce1f-b59e-47cc-9015-429f8a7858cd-ovnkube-script-lib\") pod \"ovnkube-node-t87pw\" (UID: \"e1dcce1f-b59e-47cc-9015-429f8a7858cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-t87pw"
Jul 30 17:00:28.101972 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.101951    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9485af8e-c7d9-4b8a-ab94-aa3b30415381-agent-certs\") pod \"konnectivity-agent-8lckj\" (UID: \"9485af8e-c7d9-4b8a-ab94-aa3b30415381\") " pod="kube-system/konnectivity-agent-8lckj"
Jul 30 17:00:28.107139 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.107113    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd6ws\" (UniqueName: \"kubernetes.io/projected/a2093549-0eca-49bd-96d3-2e0c01a4c19c-kube-api-access-nd6ws\") pod \"multus-nfpq4\" (UID: \"a2093549-0eca-49bd-96d3-2e0c01a4c19c\") " pod="openshift-multus/multus-nfpq4"
Jul 30 17:00:28.107591 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.107541    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv555\" (UniqueName: \"kubernetes.io/projected/97f5f5b8-ce32-41cd-9b5c-c94f2ae6bbfe-kube-api-access-vv555\") pod \"node-ca-xglts\" (UID: \"97f5f5b8-ce32-41cd-9b5c-c94f2ae6bbfe\") " pod="openshift-image-registry/node-ca-xglts"
Jul 30 17:00:28.107895 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.107871    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhhj5\" (UniqueName: \"kubernetes.io/projected/9fcdb3a5-74bc-44be-993f-fbc90e5db202-kube-api-access-mhhj5\") pod \"multus-additional-cni-plugins-qbp2g\" (UID: \"9fcdb3a5-74bc-44be-993f-fbc90e5db202\") " pod="openshift-multus/multus-additional-cni-plugins-qbp2g"
Jul 30 17:00:28.109499 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.109425    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59lv4\" (UniqueName: \"kubernetes.io/projected/ed0f2ced-c500-4288-ac35-3e5599c9d25f-kube-api-access-59lv4\") pod \"iptables-alerter-2tdqf\" (UID: \"ed0f2ced-c500-4288-ac35-3e5599c9d25f\") " pod="openshift-network-operator/iptables-alerter-2tdqf"
Jul 30 17:00:28.109803 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.109780    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfkrg\" (UniqueName: \"kubernetes.io/projected/a28bc718-4b4f-408b-b1ca-57f116295c99-kube-api-access-mfkrg\") pod \"aws-ebs-csi-driver-node-9hmcr\" (UID: \"a28bc718-4b4f-408b-b1ca-57f116295c99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9hmcr"
Jul 30 17:00:28.110277 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.110240    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgmjc\" (UniqueName: \"kubernetes.io/projected/3534e135-e217-49b8-8866-8cd33104b77e-kube-api-access-kgmjc\") pod \"node-resolver-97k9j\" (UID: \"3534e135-e217-49b8-8866-8cd33104b77e\") " pod="openshift-dns/node-resolver-97k9j"
Jul 30 17:00:28.110459 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.110340    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bzxp\" (UniqueName: \"kubernetes.io/projected/e1dcce1f-b59e-47cc-9015-429f8a7858cd-kube-api-access-6bzxp\") pod \"ovnkube-node-t87pw\" (UID: \"e1dcce1f-b59e-47cc-9015-429f8a7858cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-t87pw"
Jul 30 17:00:28.187779 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.187740    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bncwx"
Jul 30 17:00:28.194526 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.194502    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xglts"
Jul 30 17:00:28.203618 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.203592    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-97k9j"
Jul 30 17:00:28.210605 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.210527    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-t87pw"
Jul 30 17:00:28.217530 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.217511    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qbp2g"
Jul 30 17:00:28.224362 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.224342    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nfpq4"
Jul 30 17:00:28.231239 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.231217    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8lckj"
Jul 30 17:00:28.240185 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.240160    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2tdqf"
Jul 30 17:00:28.246233 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.246210    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9hmcr"
Jul 30 17:00:28.408534 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.408441    2555 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 30 17:00:28.502433 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.502398    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmqln\" (UniqueName: \"kubernetes.io/projected/fc0a3112-260e-47c8-87fd-bf3666ea17d0-kube-api-access-rmqln\") pod \"network-check-target-qvgrv\" (UID: \"fc0a3112-260e-47c8-87fd-bf3666ea17d0\") " pod="openshift-network-diagnostics/network-check-target-qvgrv"
Jul 30 17:00:28.502619 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.502439    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/51afa6bb-724f-47ab-b47b-36dc8a61ca25-original-pull-secret\") pod \"global-pull-secret-syncer-9hlj5\" (UID: \"51afa6bb-724f-47ab-b47b-36dc8a61ca25\") " pod="kube-system/global-pull-secret-syncer-9hlj5"
Jul 30 17:00:28.502619 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.502478    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e85f9d71-fe3d-49d8-93e2-e4ac101603c7-metrics-certs\") pod \"network-metrics-daemon-55hfh\" (UID: \"e85f9d71-fe3d-49d8-93e2-e4ac101603c7\") " pod="openshift-multus/network-metrics-daemon-55hfh"
Jul 30 17:00:28.502619 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:28.502577    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 30 17:00:28.502619 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:28.502601    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 30 17:00:28.502619 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:28.502609    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 17:00:28.502619 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:28.502615    2555 projected.go:196] Error preparing data for projected volume kube-api-access-rmqln for pod openshift-network-diagnostics/network-check-target-qvgrv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 17:00:28.502931 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:28.502626    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 30 17:00:28.502931 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:28.502675    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e85f9d71-fe3d-49d8-93e2-e4ac101603c7-metrics-certs podName:e85f9d71-fe3d-49d8-93e2-e4ac101603c7 nodeName:}" failed. No retries permitted until 2026-07-30 17:00:29.502658276 +0000 UTC m=+3.990025660 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e85f9d71-fe3d-49d8-93e2-e4ac101603c7-metrics-certs") pod "network-metrics-daemon-55hfh" (UID: "e85f9d71-fe3d-49d8-93e2-e4ac101603c7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 17:00:28.502931 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:28.502691    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51afa6bb-724f-47ab-b47b-36dc8a61ca25-original-pull-secret podName:51afa6bb-724f-47ab-b47b-36dc8a61ca25 nodeName:}" failed. No retries permitted until 2026-07-30 17:00:29.502684319 +0000 UTC m=+3.990051699 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/51afa6bb-724f-47ab-b47b-36dc8a61ca25-original-pull-secret") pod "global-pull-secret-syncer-9hlj5" (UID: "51afa6bb-724f-47ab-b47b-36dc8a61ca25") : object "kube-system"/"original-pull-secret" not registered
Jul 30 17:00:28.502931 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:28.502705    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fc0a3112-260e-47c8-87fd-bf3666ea17d0-kube-api-access-rmqln podName:fc0a3112-260e-47c8-87fd-bf3666ea17d0 nodeName:}" failed. No retries permitted until 2026-07-30 17:00:29.502698958 +0000 UTC m=+3.990066338 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-rmqln" (UniqueName: "kubernetes.io/projected/fc0a3112-260e-47c8-87fd-bf3666ea17d0-kube-api-access-rmqln") pod "network-check-target-qvgrv" (UID: "fc0a3112-260e-47c8-87fd-bf3666ea17d0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 17:00:28.838629 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:28.838594    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fcdb3a5_74bc_44be_993f_fbc90e5db202.slice/crio-056592b46838768c79981e7ec9ec74159ac4ae9e83722092b909019c9ff583b7 WatchSource:0}: Error finding container 056592b46838768c79981e7ec9ec74159ac4ae9e83722092b909019c9ff583b7: Status 404 returned error can't find the container with id 056592b46838768c79981e7ec9ec74159ac4ae9e83722092b909019c9ff583b7
Jul 30 17:00:28.934350 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.934278    2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-29 16:55:28 +0000 UTC" deadline="2028-05-13 08:49:19.789183352 +0000 UTC"
Jul 30 17:00:28.934350 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:28.934346    2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15663h48m50.85484234s"
Jul 30 17:00:29.064341 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:29.064302    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qbp2g" event={"ID":"9fcdb3a5-74bc-44be-993f-fbc90e5db202","Type":"ContainerStarted","Data":"056592b46838768c79981e7ec9ec74159ac4ae9e83722092b909019c9ff583b7"}
Jul 30 17:00:29.179923 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:29.179787    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1dcce1f_b59e_47cc_9015_429f8a7858cd.slice/crio-4d032858f6c4a1eac18549904ce12d9b1cb8ec1d843e97f38385501b7ffa17c6 WatchSource:0}: Error finding container 4d032858f6c4a1eac18549904ce12d9b1cb8ec1d843e97f38385501b7ffa17c6: Status 404 returned error can't find the container with id 4d032858f6c4a1eac18549904ce12d9b1cb8ec1d843e97f38385501b7ffa17c6
Jul 30 17:00:29.186216 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:29.186183    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2093549_0eca_49bd_96d3_2e0c01a4c19c.slice/crio-0668aeac0a143efbcc974d99199c64eb72ce89fc9b9b0bac35f14baba38d8f09 WatchSource:0}: Error finding container 0668aeac0a143efbcc974d99199c64eb72ce89fc9b9b0bac35f14baba38d8f09: Status 404 returned error can't find the container with id 0668aeac0a143efbcc974d99199c64eb72ce89fc9b9b0bac35f14baba38d8f09
Jul 30 17:00:29.187757 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:29.187708    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9485af8e_c7d9_4b8a_ab94_aa3b30415381.slice/crio-d346738345bcaf564790d7f7a9940dd4c38e9547f320fd103fee76f67627c7cd WatchSource:0}: Error finding container d346738345bcaf564790d7f7a9940dd4c38e9547f320fd103fee76f67627c7cd: Status 404 returned error can't find the container with id d346738345bcaf564790d7f7a9940dd4c38e9547f320fd103fee76f67627c7cd
Jul 30 17:00:29.189124 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:29.188972    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80fefbe4_044f_4c09_b4ae_4bcc3d5684d6.slice/crio-76a6e43e2c001c5d703c4f0e45495cfc000f542bd88e4bacf0eb69e66bf14983 WatchSource:0}: Error finding container 76a6e43e2c001c5d703c4f0e45495cfc000f542bd88e4bacf0eb69e66bf14983: Status 404 returned error can't find the container with id 76a6e43e2c001c5d703c4f0e45495cfc000f542bd88e4bacf0eb69e66bf14983
Jul 30 17:00:29.191096 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:29.190467    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97f5f5b8_ce32_41cd_9b5c_c94f2ae6bbfe.slice/crio-5ff3ddb778b71238e57f5ffea515323f006b26cdc1b1e19b60096012ca7dd391 WatchSource:0}: Error finding container 5ff3ddb778b71238e57f5ffea515323f006b26cdc1b1e19b60096012ca7dd391: Status 404 returned error can't find the container with id 5ff3ddb778b71238e57f5ffea515323f006b26cdc1b1e19b60096012ca7dd391
Jul 30 17:00:29.191096 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:29.190815    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3534e135_e217_49b8_8866_8cd33104b77e.slice/crio-e73aa32526dd2c80f628f278a080774331e66435b91cde657858b31af9cc0f72 WatchSource:0}: Error finding container e73aa32526dd2c80f628f278a080774331e66435b91cde657858b31af9cc0f72: Status 404 returned error can't find the container with id e73aa32526dd2c80f628f278a080774331e66435b91cde657858b31af9cc0f72
Jul 30 17:00:29.191871 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:00:29.191759    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda28bc718_4b4f_408b_b1ca_57f116295c99.slice/crio-ead272e4a76dfe334d60356bc9b6e1c5af6d8f343221e57a4c4f84f19e63732c WatchSource:0}: Error finding container ead272e4a76dfe334d60356bc9b6e1c5af6d8f343221e57a4c4f84f19e63732c: Status 404 returned error can't find the container with id ead272e4a76dfe334d60356bc9b6e1c5af6d8f343221e57a4c4f84f19e63732c
Jul 30 17:00:29.510883 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:29.510666    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e85f9d71-fe3d-49d8-93e2-e4ac101603c7-metrics-certs\") pod \"network-metrics-daemon-55hfh\" (UID: \"e85f9d71-fe3d-49d8-93e2-e4ac101603c7\") " pod="openshift-multus/network-metrics-daemon-55hfh"
Jul 30 17:00:29.511044 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:29.510916    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmqln\" (UniqueName: \"kubernetes.io/projected/fc0a3112-260e-47c8-87fd-bf3666ea17d0-kube-api-access-rmqln\") pod \"network-check-target-qvgrv\" (UID: \"fc0a3112-260e-47c8-87fd-bf3666ea17d0\") " pod="openshift-network-diagnostics/network-check-target-qvgrv"
Jul 30 17:00:29.511044 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:29.510832    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 17:00:29.511044 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:29.510945    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/51afa6bb-724f-47ab-b47b-36dc8a61ca25-original-pull-secret\") pod \"global-pull-secret-syncer-9hlj5\" (UID: \"51afa6bb-724f-47ab-b47b-36dc8a61ca25\") " pod="kube-system/global-pull-secret-syncer-9hlj5"
Jul 30 17:00:29.511044 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:29.510995    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e85f9d71-fe3d-49d8-93e2-e4ac101603c7-metrics-certs podName:e85f9d71-fe3d-49d8-93e2-e4ac101603c7 nodeName:}" failed. No retries permitted until 2026-07-30 17:00:31.510979019 +0000 UTC m=+5.998346401 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e85f9d71-fe3d-49d8-93e2-e4ac101603c7-metrics-certs") pod "network-metrics-daemon-55hfh" (UID: "e85f9d71-fe3d-49d8-93e2-e4ac101603c7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 17:00:29.511044 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:29.511044    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 30 17:00:29.511322 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:29.511083    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 30 17:00:29.511322 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:29.511106    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 30 17:00:29.511322 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:29.511114    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51afa6bb-724f-47ab-b47b-36dc8a61ca25-original-pull-secret podName:51afa6bb-724f-47ab-b47b-36dc8a61ca25 nodeName:}" failed. No retries permitted until 2026-07-30 17:00:31.511097488 +0000 UTC m=+5.998464870 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/51afa6bb-724f-47ab-b47b-36dc8a61ca25-original-pull-secret") pod "global-pull-secret-syncer-9hlj5" (UID: "51afa6bb-724f-47ab-b47b-36dc8a61ca25") : object "kube-system"/"original-pull-secret" not registered
Jul 30 17:00:29.511322 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:29.511121    2555 projected.go:196] Error preparing data for projected volume kube-api-access-rmqln for pod openshift-network-diagnostics/network-check-target-qvgrv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 17:00:29.511322 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:29.511169    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fc0a3112-260e-47c8-87fd-bf3666ea17d0-kube-api-access-rmqln podName:fc0a3112-260e-47c8-87fd-bf3666ea17d0 nodeName:}" failed. No retries permitted until 2026-07-30 17:00:31.511153445 +0000 UTC m=+5.998520829 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-rmqln" (UniqueName: "kubernetes.io/projected/fc0a3112-260e-47c8-87fd-bf3666ea17d0-kube-api-access-rmqln") pod "network-check-target-qvgrv" (UID: "fc0a3112-260e-47c8-87fd-bf3666ea17d0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 17:00:29.935508 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:29.935441    2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-29 16:55:28 +0000 UTC" deadline="2028-02-25 20:07:42.319854224 +0000 UTC"
Jul 30 17:00:29.935508 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:29.935482    2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13803h7m12.384377935s"
Jul 30 17:00:30.057211 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:30.056321    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvgrv"
Jul 30 17:00:30.057211 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:30.056462    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvgrv" podUID="fc0a3112-260e-47c8-87fd-bf3666ea17d0"
Jul 30 17:00:30.057211 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:30.056903    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hlj5"
Jul 30 17:00:30.057211 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:30.056996    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9hlj5" podUID="51afa6bb-724f-47ab-b47b-36dc8a61ca25"
Jul 30 17:00:30.057211 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:30.057056    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-55hfh"
Jul 30 17:00:30.057211 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:30.057164    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-55hfh" podUID="e85f9d71-fe3d-49d8-93e2-e4ac101603c7"
Jul 30 17:00:30.074609 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:30.074480    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nfpq4" event={"ID":"a2093549-0eca-49bd-96d3-2e0c01a4c19c","Type":"ContainerStarted","Data":"0668aeac0a143efbcc974d99199c64eb72ce89fc9b9b0bac35f14baba38d8f09"}
Jul 30 17:00:30.076286 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:30.076199    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9hmcr" event={"ID":"a28bc718-4b4f-408b-b1ca-57f116295c99","Type":"ContainerStarted","Data":"ead272e4a76dfe334d60356bc9b6e1c5af6d8f343221e57a4c4f84f19e63732c"}
Jul 30 17:00:30.078875 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:30.078804    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2tdqf" event={"ID":"ed0f2ced-c500-4288-ac35-3e5599c9d25f","Type":"ContainerStarted","Data":"c088fe7ee4aa6c478b8f7c42a05d86397310d2982e2f6d8470f6387986cc2a2b"}
Jul 30 17:00:30.084265 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:30.084148    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t87pw" event={"ID":"e1dcce1f-b59e-47cc-9015-429f8a7858cd","Type":"ContainerStarted","Data":"4d032858f6c4a1eac18549904ce12d9b1cb8ec1d843e97f38385501b7ffa17c6"}
Jul 30 17:00:30.090342 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:30.090313    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-190.ec2.internal" event={"ID":"041d73f2cda3ec42dcee6249e2df6511","Type":"ContainerStarted","Data":"f46e95085475c228079a4980e00515da28ce8a7edc49f74746320e2b1c0ca3d9"}
Jul 30 17:00:30.104240 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:30.104206    2555 generic.go:356] "Generic (PLEG): container finished" podID="3e02b1d2169088ae669788950ba2bc8f" containerID="7599c59f718adac7cb6a85372a0b289be31e27399926b29aabea47fc8eb4bb23" exitCode=0
Jul 30 17:00:30.104529 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:30.104466    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-190.ec2.internal" event={"ID":"3e02b1d2169088ae669788950ba2bc8f","Type":"ContainerDied","Data":"7599c59f718adac7cb6a85372a0b289be31e27399926b29aabea47fc8eb4bb23"}
Jul 30 17:00:30.109530 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:30.108253    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-190.ec2.internal" podStartSLOduration=3.108233605 podStartE2EDuration="3.108233605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 17:00:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:00:30.107284253 +0000 UTC m=+4.594651669" watchObservedRunningTime="2026-07-30 17:00:30.108233605 +0000 UTC m=+4.595601016"
Jul 30 17:00:30.118653 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:30.118546    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-97k9j" event={"ID":"3534e135-e217-49b8-8866-8cd33104b77e","Type":"ContainerStarted","Data":"e73aa32526dd2c80f628f278a080774331e66435b91cde657858b31af9cc0f72"}
Jul 30 17:00:30.125758 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:30.125115    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bncwx" event={"ID":"80fefbe4-044f-4c09-b4ae-4bcc3d5684d6","Type":"ContainerStarted","Data":"76a6e43e2c001c5d703c4f0e45495cfc000f542bd88e4bacf0eb69e66bf14983"}
Jul 30 17:00:30.137141 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:30.133839    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8lckj" event={"ID":"9485af8e-c7d9-4b8a-ab94-aa3b30415381","Type":"ContainerStarted","Data":"d346738345bcaf564790d7f7a9940dd4c38e9547f320fd103fee76f67627c7cd"}
Jul 30 17:00:30.154331 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:30.154286    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xglts" event={"ID":"97f5f5b8-ce32-41cd-9b5c-c94f2ae6bbfe","Type":"ContainerStarted","Data":"5ff3ddb778b71238e57f5ffea515323f006b26cdc1b1e19b60096012ca7dd391"}
Jul 30 17:00:31.183523 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:31.183485    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-190.ec2.internal" event={"ID":"3e02b1d2169088ae669788950ba2bc8f","Type":"ContainerStarted","Data":"57a9821e153527d05a530a65093832cbead6d95b01860ac3550c4f4fd316e330"}
Jul 30 17:00:31.530066 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:31.529952    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmqln\" (UniqueName: \"kubernetes.io/projected/fc0a3112-260e-47c8-87fd-bf3666ea17d0-kube-api-access-rmqln\") pod \"network-check-target-qvgrv\" (UID: \"fc0a3112-260e-47c8-87fd-bf3666ea17d0\") " pod="openshift-network-diagnostics/network-check-target-qvgrv"
Jul 30 17:00:31.530066 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:31.530008    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/51afa6bb-724f-47ab-b47b-36dc8a61ca25-original-pull-secret\") pod \"global-pull-secret-syncer-9hlj5\" (UID: \"51afa6bb-724f-47ab-b47b-36dc8a61ca25\") " pod="kube-system/global-pull-secret-syncer-9hlj5"
Jul 30 17:00:31.530301 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:31.530069    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e85f9d71-fe3d-49d8-93e2-e4ac101603c7-metrics-certs\") pod \"network-metrics-daemon-55hfh\" (UID: \"e85f9d71-fe3d-49d8-93e2-e4ac101603c7\") " pod="openshift-multus/network-metrics-daemon-55hfh"
Jul 30 17:00:31.530301 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:31.530228    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 17:00:31.530301 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:31.530293    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e85f9d71-fe3d-49d8-93e2-e4ac101603c7-metrics-certs podName:e85f9d71-fe3d-49d8-93e2-e4ac101603c7 nodeName:}" failed. No retries permitted until 2026-07-30 17:00:35.530273403 +0000 UTC m=+10.017640786 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e85f9d71-fe3d-49d8-93e2-e4ac101603c7-metrics-certs") pod "network-metrics-daemon-55hfh" (UID: "e85f9d71-fe3d-49d8-93e2-e4ac101603c7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 17:00:31.530534 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:31.530384    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 30 17:00:31.530534 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:31.530401    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 30 17:00:31.530534 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:31.530413    2555 projected.go:196] Error preparing data for projected volume kube-api-access-rmqln for pod openshift-network-diagnostics/network-check-target-qvgrv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 17:00:31.530534 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:31.530454    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fc0a3112-260e-47c8-87fd-bf3666ea17d0-kube-api-access-rmqln podName:fc0a3112-260e-47c8-87fd-bf3666ea17d0 nodeName:}" failed. No retries permitted until 2026-07-30 17:00:35.530440505 +0000 UTC m=+10.017807890 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-rmqln" (UniqueName: "kubernetes.io/projected/fc0a3112-260e-47c8-87fd-bf3666ea17d0-kube-api-access-rmqln") pod "network-check-target-qvgrv" (UID: "fc0a3112-260e-47c8-87fd-bf3666ea17d0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 17:00:31.530534 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:31.530512    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 30 17:00:31.530798 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:31.530544    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51afa6bb-724f-47ab-b47b-36dc8a61ca25-original-pull-secret podName:51afa6bb-724f-47ab-b47b-36dc8a61ca25 nodeName:}" failed. No retries permitted until 2026-07-30 17:00:35.530533884 +0000 UTC m=+10.017901265 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/51afa6bb-724f-47ab-b47b-36dc8a61ca25-original-pull-secret") pod "global-pull-secret-syncer-9hlj5" (UID: "51afa6bb-724f-47ab-b47b-36dc8a61ca25") : object "kube-system"/"original-pull-secret" not registered
Jul 30 17:00:32.054901 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:32.054822    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvgrv"
Jul 30 17:00:32.054901 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:32.054834    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hlj5"
Jul 30 17:00:32.055187 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:32.054949    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvgrv" podUID="fc0a3112-260e-47c8-87fd-bf3666ea17d0"
Jul 30 17:00:32.055187 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:32.055042    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9hlj5" podUID="51afa6bb-724f-47ab-b47b-36dc8a61ca25"
Jul 30 17:00:32.055187 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:32.055120    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-55hfh"
Jul 30 17:00:32.055361 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:32.055213    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-55hfh" podUID="e85f9d71-fe3d-49d8-93e2-e4ac101603c7"
Jul 30 17:00:34.055208 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:34.055110    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-55hfh"
Jul 30 17:00:34.055208 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:34.055149    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hlj5"
Jul 30 17:00:34.055613 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:34.055110    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvgrv"
Jul 30 17:00:34.055613 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:34.055287    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-55hfh" podUID="e85f9d71-fe3d-49d8-93e2-e4ac101603c7"
Jul 30 17:00:34.055613 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:34.055371    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvgrv" podUID="fc0a3112-260e-47c8-87fd-bf3666ea17d0"
Jul 30 17:00:34.055613 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:34.055438    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9hlj5" podUID="51afa6bb-724f-47ab-b47b-36dc8a61ca25"
Jul 30 17:00:35.563727 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:35.563683    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e85f9d71-fe3d-49d8-93e2-e4ac101603c7-metrics-certs\") pod \"network-metrics-daemon-55hfh\" (UID: \"e85f9d71-fe3d-49d8-93e2-e4ac101603c7\") " pod="openshift-multus/network-metrics-daemon-55hfh"
Jul 30 17:00:35.564230 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:35.563774    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmqln\" (UniqueName: \"kubernetes.io/projected/fc0a3112-260e-47c8-87fd-bf3666ea17d0-kube-api-access-rmqln\") pod \"network-check-target-qvgrv\" (UID: \"fc0a3112-260e-47c8-87fd-bf3666ea17d0\") " pod="openshift-network-diagnostics/network-check-target-qvgrv"
Jul 30 17:00:35.564230 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:35.563802    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/51afa6bb-724f-47ab-b47b-36dc8a61ca25-original-pull-secret\") pod \"global-pull-secret-syncer-9hlj5\" (UID: \"51afa6bb-724f-47ab-b47b-36dc8a61ca25\") " pod="kube-system/global-pull-secret-syncer-9hlj5"
Jul 30 17:00:35.564230 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:35.563877    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 17:00:35.564230 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:35.563910    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 30 17:00:35.564230 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:35.563949    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 30 17:00:35.564230 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:35.563972    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 30 17:00:35.564230 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:35.563987    2555 projected.go:196] Error preparing data for projected volume kube-api-access-rmqln for pod openshift-network-diagnostics/network-check-target-qvgrv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 17:00:35.564230 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:35.563959    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e85f9d71-fe3d-49d8-93e2-e4ac101603c7-metrics-certs podName:e85f9d71-fe3d-49d8-93e2-e4ac101603c7 nodeName:}" failed. No retries permitted until 2026-07-30 17:00:43.563937915 +0000 UTC m=+18.051305298 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e85f9d71-fe3d-49d8-93e2-e4ac101603c7-metrics-certs") pod "network-metrics-daemon-55hfh" (UID: "e85f9d71-fe3d-49d8-93e2-e4ac101603c7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 17:00:35.564230 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:35.564065    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51afa6bb-724f-47ab-b47b-36dc8a61ca25-original-pull-secret podName:51afa6bb-724f-47ab-b47b-36dc8a61ca25 nodeName:}" failed. No retries permitted until 2026-07-30 17:00:43.564047547 +0000 UTC m=+18.051414926 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/51afa6bb-724f-47ab-b47b-36dc8a61ca25-original-pull-secret") pod "global-pull-secret-syncer-9hlj5" (UID: "51afa6bb-724f-47ab-b47b-36dc8a61ca25") : object "kube-system"/"original-pull-secret" not registered
Jul 30 17:00:35.564230 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:35.564080    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fc0a3112-260e-47c8-87fd-bf3666ea17d0-kube-api-access-rmqln podName:fc0a3112-260e-47c8-87fd-bf3666ea17d0 nodeName:}" failed. No retries permitted until 2026-07-30 17:00:43.564072169 +0000 UTC m=+18.051439547 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-rmqln" (UniqueName: "kubernetes.io/projected/fc0a3112-260e-47c8-87fd-bf3666ea17d0-kube-api-access-rmqln") pod "network-check-target-qvgrv" (UID: "fc0a3112-260e-47c8-87fd-bf3666ea17d0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 17:00:36.055617 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:36.055539    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvgrv"
Jul 30 17:00:36.055777 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:36.055646    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvgrv" podUID="fc0a3112-260e-47c8-87fd-bf3666ea17d0"
Jul 30 17:00:36.056186 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:36.056053    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hlj5"
Jul 30 17:00:36.056186 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:36.056148    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9hlj5" podUID="51afa6bb-724f-47ab-b47b-36dc8a61ca25"
Jul 30 17:00:36.056591 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:36.056444    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-55hfh"
Jul 30 17:00:36.056591 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:36.056557    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-55hfh" podUID="e85f9d71-fe3d-49d8-93e2-e4ac101603c7"
Jul 30 17:00:38.055133 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:38.055098    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvgrv"
Jul 30 17:00:38.055603 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:38.055222    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvgrv" podUID="fc0a3112-260e-47c8-87fd-bf3666ea17d0"
Jul 30 17:00:38.055603 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:38.055098    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-55hfh"
Jul 30 17:00:38.055603 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:38.055336    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-55hfh" podUID="e85f9d71-fe3d-49d8-93e2-e4ac101603c7"
Jul 30 17:00:38.055603 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:38.055098    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hlj5"
Jul 30 17:00:38.055603 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:38.055411    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9hlj5" podUID="51afa6bb-724f-47ab-b47b-36dc8a61ca25"
Jul 30 17:00:40.055096 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:40.055020    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-55hfh"
Jul 30 17:00:40.055096 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:40.055052    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hlj5"
Jul 30 17:00:40.055096 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:40.055052    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvgrv"
Jul 30 17:00:40.055607 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:40.055203    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-55hfh" podUID="e85f9d71-fe3d-49d8-93e2-e4ac101603c7"
Jul 30 17:00:40.055607 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:40.055338    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9hlj5" podUID="51afa6bb-724f-47ab-b47b-36dc8a61ca25"
Jul 30 17:00:40.055607 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:40.055407    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvgrv" podUID="fc0a3112-260e-47c8-87fd-bf3666ea17d0"
Jul 30 17:00:42.055197 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:42.055163    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hlj5"
Jul 30 17:00:42.055197 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:42.055187    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-55hfh"
Jul 30 17:00:42.055684 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:42.055164    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvgrv"
Jul 30 17:00:42.055684 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:42.055292    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9hlj5" podUID="51afa6bb-724f-47ab-b47b-36dc8a61ca25"
Jul 30 17:00:42.055684 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:42.055362    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvgrv" podUID="fc0a3112-260e-47c8-87fd-bf3666ea17d0"
Jul 30 17:00:42.055684 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:42.055453    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-55hfh" podUID="e85f9d71-fe3d-49d8-93e2-e4ac101603c7"
Jul 30 17:00:43.621902 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:43.621868    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmqln\" (UniqueName: \"kubernetes.io/projected/fc0a3112-260e-47c8-87fd-bf3666ea17d0-kube-api-access-rmqln\") pod \"network-check-target-qvgrv\" (UID: \"fc0a3112-260e-47c8-87fd-bf3666ea17d0\") " pod="openshift-network-diagnostics/network-check-target-qvgrv"
Jul 30 17:00:43.621902 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:43.621905    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/51afa6bb-724f-47ab-b47b-36dc8a61ca25-original-pull-secret\") pod \"global-pull-secret-syncer-9hlj5\" (UID: \"51afa6bb-724f-47ab-b47b-36dc8a61ca25\") " pod="kube-system/global-pull-secret-syncer-9hlj5"
Jul 30 17:00:43.622323 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:43.621934    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e85f9d71-fe3d-49d8-93e2-e4ac101603c7-metrics-certs\") pod \"network-metrics-daemon-55hfh\" (UID: \"e85f9d71-fe3d-49d8-93e2-e4ac101603c7\") " pod="openshift-multus/network-metrics-daemon-55hfh"
Jul 30 17:00:43.622323 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:43.622052    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 17:00:43.622323 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:43.622107    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e85f9d71-fe3d-49d8-93e2-e4ac101603c7-metrics-certs podName:e85f9d71-fe3d-49d8-93e2-e4ac101603c7 nodeName:}" failed. No retries permitted until 2026-07-30 17:00:59.62209313 +0000 UTC m=+34.109460509 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e85f9d71-fe3d-49d8-93e2-e4ac101603c7-metrics-certs") pod "network-metrics-daemon-55hfh" (UID: "e85f9d71-fe3d-49d8-93e2-e4ac101603c7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 17:00:43.622323 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:43.622052    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 30 17:00:43.622323 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:43.622157    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 30 17:00:43.622323 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:43.622171    2555 projected.go:196] Error preparing data for projected volume kube-api-access-rmqln for pod openshift-network-diagnostics/network-check-target-qvgrv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 17:00:43.622323 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:43.622052    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 30 17:00:43.622323 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:43.622211    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fc0a3112-260e-47c8-87fd-bf3666ea17d0-kube-api-access-rmqln podName:fc0a3112-260e-47c8-87fd-bf3666ea17d0 nodeName:}" failed. No retries permitted until 2026-07-30 17:00:59.622198617 +0000 UTC m=+34.109566017 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-rmqln" (UniqueName: "kubernetes.io/projected/fc0a3112-260e-47c8-87fd-bf3666ea17d0-kube-api-access-rmqln") pod "network-check-target-qvgrv" (UID: "fc0a3112-260e-47c8-87fd-bf3666ea17d0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 17:00:43.622323 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:43.622292    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51afa6bb-724f-47ab-b47b-36dc8a61ca25-original-pull-secret podName:51afa6bb-724f-47ab-b47b-36dc8a61ca25 nodeName:}" failed. No retries permitted until 2026-07-30 17:00:59.622272279 +0000 UTC m=+34.109639658 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/51afa6bb-724f-47ab-b47b-36dc8a61ca25-original-pull-secret") pod "global-pull-secret-syncer-9hlj5" (UID: "51afa6bb-724f-47ab-b47b-36dc8a61ca25") : object "kube-system"/"original-pull-secret" not registered
Jul 30 17:00:44.055140 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:44.055106    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-55hfh"
Jul 30 17:00:44.055307 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:44.055224    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-55hfh" podUID="e85f9d71-fe3d-49d8-93e2-e4ac101603c7"
Jul 30 17:00:44.055405 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:44.055387    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvgrv"
Jul 30 17:00:44.055525 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:44.055501    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvgrv" podUID="fc0a3112-260e-47c8-87fd-bf3666ea17d0"
Jul 30 17:00:44.055603 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:44.055556    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hlj5"
Jul 30 17:00:44.055692 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:44.055665    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9hlj5" podUID="51afa6bb-724f-47ab-b47b-36dc8a61ca25"
Jul 30 17:00:46.055793 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:46.055757    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hlj5"
Jul 30 17:00:46.056220 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:46.055850    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9hlj5" podUID="51afa6bb-724f-47ab-b47b-36dc8a61ca25"
Jul 30 17:00:46.056220 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:46.055905    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-55hfh"
Jul 30 17:00:46.056220 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:46.056040    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-55hfh" podUID="e85f9d71-fe3d-49d8-93e2-e4ac101603c7"
Jul 30 17:00:46.056220 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:46.056088    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvgrv"
Jul 30 17:00:46.056220 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:46.056146    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvgrv" podUID="fc0a3112-260e-47c8-87fd-bf3666ea17d0"
Jul 30 17:00:48.055542 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:48.055237    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hlj5"
Jul 30 17:00:48.056234 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:48.055237    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-55hfh"
Jul 30 17:00:48.056234 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:48.055619    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9hlj5" podUID="51afa6bb-724f-47ab-b47b-36dc8a61ca25"
Jul 30 17:00:48.056234 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:48.055237    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvgrv"
Jul 30 17:00:48.056234 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:48.055681    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-55hfh" podUID="e85f9d71-fe3d-49d8-93e2-e4ac101603c7"
Jul 30 17:00:48.056234 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:48.055744    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvgrv" podUID="fc0a3112-260e-47c8-87fd-bf3666ea17d0"
Jul 30 17:00:48.218330 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:48.218300    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8lckj" event={"ID":"9485af8e-c7d9-4b8a-ab94-aa3b30415381","Type":"ContainerStarted","Data":"31809bf737f4a637cfa4c17405cbdc4dd68e4ee7c081bc6d47588c50682acf03"}
Jul 30 17:00:48.219843 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:48.219821    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xglts" event={"ID":"97f5f5b8-ce32-41cd-9b5c-c94f2ae6bbfe","Type":"ContainerStarted","Data":"4d31e8da3a9fb27302ba861729a616b558d70965782d2f7efd3ad82962dff293"}
Jul 30 17:00:48.221342 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:48.221317    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nfpq4" event={"ID":"a2093549-0eca-49bd-96d3-2e0c01a4c19c","Type":"ContainerStarted","Data":"4f70779e8aa4bdfa166786c4b3f6035a27c1e790e406bee456d2f69e3f48253d"}
Jul 30 17:00:48.222744 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:48.222725    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9hmcr" event={"ID":"a28bc718-4b4f-408b-b1ca-57f116295c99","Type":"ContainerStarted","Data":"46613da9984ca8dc18c5d0f2092370e84bc477c2fb452d8301056bcf437470b3"}
Jul 30 17:00:48.225194 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:48.225174    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t87pw" event={"ID":"e1dcce1f-b59e-47cc-9015-429f8a7858cd","Type":"ContainerStarted","Data":"950ee35cd3f039eeb94a98f32f42e79a4f317e777235e6509b55666233e774d9"}
Jul 30 17:00:48.225286 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:48.225225    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t87pw" event={"ID":"e1dcce1f-b59e-47cc-9015-429f8a7858cd","Type":"ContainerStarted","Data":"3dd97c2c141600edd2bdba90f909425395c59db7415ac81c4ce1c5a865b083b4"}
Jul 30 17:00:48.225286 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:48.225256    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t87pw" event={"ID":"e1dcce1f-b59e-47cc-9015-429f8a7858cd","Type":"ContainerStarted","Data":"8c797b25df681f752c8e348c4588c08bfa454b3f59aa61911c67d68b8a5d57c8"}
Jul 30 17:00:48.225286 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:48.225284    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t87pw" event={"ID":"e1dcce1f-b59e-47cc-9015-429f8a7858cd","Type":"ContainerStarted","Data":"2252181bae2beffd3269b596accaffb2e41dfe68ac6b86d7860b76532e65816a"}
Jul 30 17:00:48.225414 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:48.225314    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t87pw" event={"ID":"e1dcce1f-b59e-47cc-9015-429f8a7858cd","Type":"ContainerStarted","Data":"e3acd70c71df0f0935b9f865d8c935489bbc723a9b7ecabbbbcc20b68d7f8e82"}
Jul 30 17:00:48.226387 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:48.226369    2555 generic.go:356] "Generic (PLEG): container finished" podID="9fcdb3a5-74bc-44be-993f-fbc90e5db202" containerID="59d2c818aa93ebcd29b69a8e4bd7828e68c4b7240a87c2a27fcc973e20b139df" exitCode=0
Jul 30 17:00:48.226467 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:48.226428    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qbp2g" event={"ID":"9fcdb3a5-74bc-44be-993f-fbc90e5db202","Type":"ContainerDied","Data":"59d2c818aa93ebcd29b69a8e4bd7828e68c4b7240a87c2a27fcc973e20b139df"}
Jul 30 17:00:48.227816 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:48.227798    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-97k9j" event={"ID":"3534e135-e217-49b8-8866-8cd33104b77e","Type":"ContainerStarted","Data":"d7537070207f555a64db76461446bc34528649397e05f66cb63d33c76e4f4bc0"}
Jul 30 17:00:48.229352 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:48.229336    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bncwx" event={"ID":"80fefbe4-044f-4c09-b4ae-4bcc3d5684d6","Type":"ContainerStarted","Data":"1ce68187759340bac7d528e511ae9b21e39b2667a4c9b6fa83f1a5e1321bff78"}
Jul 30 17:00:48.234359 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:48.234326    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-190.ec2.internal" podStartSLOduration=21.234314854 podStartE2EDuration="21.234314854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 17:00:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:00:31.203684158 +0000 UTC m=+5.691051565" watchObservedRunningTime="2026-07-30 17:00:48.234314854 +0000 UTC m=+22.721682255"
Jul 30 17:00:48.234581 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:48.234562    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-8lckj" podStartSLOduration=3.839685588 podStartE2EDuration="22.234555422s" totalImagesPullingTime="18.394869834s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:00:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:00:48.233847058 +0000 UTC m=+22.721214458" watchObservedRunningTime="2026-07-30 17:00:48.234555422 +0000 UTC m=+22.721922822"
Jul 30 17:00:48.248763 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:48.248721    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-xglts" podStartSLOduration=3.852943336 podStartE2EDuration="22.248709075s" totalImagesPullingTime="18.395765739s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 17:00:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:00:48.248505791 +0000 UTC m=+22.735873170" watchObservedRunningTime="2026-07-30 17:00:48.248709075 +0000 UTC m=+22.736076476"
Jul 30 17:00:48.295332 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:48.295284    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-bncwx" podStartSLOduration=3.895695513 podStartE2EDuration="22.295268524s" totalImagesPullingTime="18.399573011s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 17:00:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:00:48.29490208 +0000 UTC m=+22.782269480" watchObservedRunningTime="2026-07-30 17:00:48.295268524 +0000 UTC m=+22.782635955"
Jul 30 17:00:48.295502 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:48.295483    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-nfpq4" podStartSLOduration=3.842537659 podStartE2EDuration="22.29547827s" totalImagesPullingTime="18.452940611s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 17:00:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:00:48.274086333 +0000 UTC m=+22.761453733" watchObservedRunningTime="2026-07-30 17:00:48.29547827 +0000 UTC m=+22.782845670"
Jul 30 17:00:48.310729 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:48.310685    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-97k9j" podStartSLOduration=3.915968593 podStartE2EDuration="22.310671295s" totalImagesPullingTime="18.394702702s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 17:00:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:00:48.310472873 +0000 UTC m=+22.797840275" watchObservedRunningTime="2026-07-30 17:00:48.310671295 +0000 UTC m=+22.798038696"
Jul 30 17:00:48.726435 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:48.726373    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:00:48.860398 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:48.860272    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:00:48.961343 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:48.961308    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:00:49.067823 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:49.067797    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:00:49.163606 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:49.163579    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:00:49.202815 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:49.202789    2555 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 30 17:00:49.234351 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:49.234317    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9hmcr" event={"ID":"a28bc718-4b4f-408b-b1ca-57f116295c99","Type":"ContainerStarted","Data":"be7325a7f0e2d8e47767f25cee024c44b4615b74b3542d185af25cb938d19cf6"}
Jul 30 17:00:49.236034 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:49.235992    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2tdqf" event={"ID":"ed0f2ced-c500-4288-ac35-3e5599c9d25f","Type":"ContainerStarted","Data":"430100a2591f5c85e6d70b149041c6a2996708a139dcf4bd4662169fc9d0f8f0"}
Jul 30 17:00:49.239556 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:49.239529    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t87pw" event={"ID":"e1dcce1f-b59e-47cc-9015-429f8a7858cd","Type":"ContainerStarted","Data":"15206f5c6979a320f55490ce61cde278e3f5e92ca333b86be73ed446a97736ee"}
Jul 30 17:00:49.251685 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:49.251637    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-2tdqf" podStartSLOduration=4.847778457 podStartE2EDuration="23.251621231s" totalImagesPullingTime="18.403842774s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 17:00:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:00:49.251199627 +0000 UTC m=+23.738567030" watchObservedRunningTime="2026-07-30 17:00:49.251621231 +0000 UTC m=+23.738988631"
Jul 30 17:00:49.443114 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:49.443032    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-8lckj"
Jul 30 17:00:49.443600 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:49.443577    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-8lckj"
Jul 30 17:00:49.845341 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:49.845303    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:00:49.947022 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:49.946834    2555 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-30T17:00:49.202811895Z","UUID":"1dc25168-7145-4b9f-b69c-455ffeee9be9","Handler":null,"Name":"","Endpoint":""}
Jul 30 17:00:49.948896 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:49.948875    2555 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 30 17:00:49.949042 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:49.948919    2555 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 30 17:00:50.055409 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:50.055378    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-55hfh"
Jul 30 17:00:50.055596 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:50.055429    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvgrv"
Jul 30 17:00:50.055596 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:50.055377    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hlj5"
Jul 30 17:00:50.055596 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:50.055574    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-55hfh" podUID="e85f9d71-fe3d-49d8-93e2-e4ac101603c7"
Jul 30 17:00:50.055726 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:50.055656    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvgrv" podUID="fc0a3112-260e-47c8-87fd-bf3666ea17d0"
Jul 30 17:00:50.055783 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:50.055750    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9hlj5" podUID="51afa6bb-724f-47ab-b47b-36dc8a61ca25"
Jul 30 17:00:50.242693 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:50.242496    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-8lckj"
Jul 30 17:00:50.243172 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:50.242823    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-8lckj"
Jul 30 17:00:50.308399 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:50.308369    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:00:50.502401 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:50.502094    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:00:50.707626 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:50.707602    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:00:50.907042 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:50.906970    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:00:51.099814 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:51.099779    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:00:51.247119 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:51.247031    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9hmcr" event={"ID":"a28bc718-4b4f-408b-b1ca-57f116295c99","Type":"ContainerStarted","Data":"6b94decc66a4127aa0a6b054254a17649b06956e04091fe0aa37cf45589ca34a"}
Jul 30 17:00:51.251206 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:51.251177    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t87pw" event={"ID":"e1dcce1f-b59e-47cc-9015-429f8a7858cd","Type":"ContainerStarted","Data":"ddfd3b204f0afb905ef07ab6928cbaa638c51df2bc659b08628fedde9cbd1073"}
Jul 30 17:00:51.304702 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:51.304669    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:00:51.367093 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:51.367066    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:00:52.055281 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:52.055254    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvgrv"
Jul 30 17:00:52.055461 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:52.055256    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-55hfh"
Jul 30 17:00:52.055461 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:52.055373    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvgrv" podUID="fc0a3112-260e-47c8-87fd-bf3666ea17d0"
Jul 30 17:00:52.055461 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:52.055266    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hlj5"
Jul 30 17:00:52.055461 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:52.055444    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-55hfh" podUID="e85f9d71-fe3d-49d8-93e2-e4ac101603c7"
Jul 30 17:00:52.055676 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:52.055598    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9hlj5" podUID="51afa6bb-724f-47ab-b47b-36dc8a61ca25"
Jul 30 17:00:52.118026 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:52.117981    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:00:52.302327 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:52.302300    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:00:52.423581 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:52.422927    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:00:52.523777 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:52.523754    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:00:53.195129 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:53.194930    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:00:53.248352 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:53.248324    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:00:53.258816 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:53.258794    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t87pw" event={"ID":"e1dcce1f-b59e-47cc-9015-429f8a7858cd","Type":"ContainerStarted","Data":"b1b9d6f564d1412214fda45eeb02a42410aa77d362af42d4d21c63bb7d981ec4"}
Jul 30 17:00:53.259067 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:53.259053    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-t87pw"
Jul 30 17:00:53.260604 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:53.260584    2555 generic.go:356] "Generic (PLEG): container finished" podID="9fcdb3a5-74bc-44be-993f-fbc90e5db202" containerID="937a835719bb3ad1e99f74633e30d5784e993b0311f85e2aa305d43abed1ad47" exitCode=0
Jul 30 17:00:53.260705 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:53.260629    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qbp2g" event={"ID":"9fcdb3a5-74bc-44be-993f-fbc90e5db202","Type":"ContainerDied","Data":"937a835719bb3ad1e99f74633e30d5784e993b0311f85e2aa305d43abed1ad47"}
Jul 30 17:00:53.275916 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:53.275895    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-t87pw"
Jul 30 17:00:53.287387 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:53.287349    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-t87pw" podStartSLOduration=8.816443678 podStartE2EDuration="27.28733737s" totalImagesPullingTime="18.470893692s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 17:00:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:00:53.286065074 +0000 UTC m=+27.773432477" watchObservedRunningTime="2026-07-30 17:00:53.28733737 +0000 UTC m=+27.774704770"
Jul 30 17:00:53.287588 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:53.287564    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9hmcr" podStartSLOduration=6.213555722 podStartE2EDuration="27.287558611s" totalImagesPullingTime="21.074002889s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 17:00:26 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:00:51.265224032 +0000 UTC m=+25.752591433" watchObservedRunningTime="2026-07-30 17:00:53.287558611 +0000 UTC m=+27.774926012"
Jul 30 17:00:53.698995 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:53.698942    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:00:53.952350 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:53.952277    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-t87pw"
Jul 30 17:00:54.054745 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:54.054714    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvgrv"
Jul 30 17:00:54.054745 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:54.054714    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hlj5"
Jul 30 17:00:54.055076 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:54.054715    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-55hfh"
Jul 30 17:00:54.055076 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:54.054816    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvgrv" podUID="fc0a3112-260e-47c8-87fd-bf3666ea17d0"
Jul 30 17:00:54.055076 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:54.054887    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-55hfh" podUID="e85f9d71-fe3d-49d8-93e2-e4ac101603c7"
Jul 30 17:00:54.055076 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:54.054945    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9hlj5" podUID="51afa6bb-724f-47ab-b47b-36dc8a61ca25"
Jul 30 17:00:54.134891 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:54.134860    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:00:54.263221 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:54.263191    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-t87pw"
Jul 30 17:00:54.278832 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:54.278809    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-t87pw"
Jul 30 17:00:54.498130 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:54.498051    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:00:54.624447 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:54.624417    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:00:54.881559 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:54.881530    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:00:54.982578 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:54.982548    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:00:55.267330 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:55.267258    2555 generic.go:356] "Generic (PLEG): container finished" podID="9fcdb3a5-74bc-44be-993f-fbc90e5db202" containerID="94f4b17b99edb840d50e2dc3d1b787fd9304fcfb0c4a59b055306351b605c694" exitCode=0
Jul 30 17:00:55.267485 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:55.267349    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qbp2g" event={"ID":"9fcdb3a5-74bc-44be-993f-fbc90e5db202","Type":"ContainerDied","Data":"94f4b17b99edb840d50e2dc3d1b787fd9304fcfb0c4a59b055306351b605c694"}
Jul 30 17:00:55.386225 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:55.386191    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:00:55.508525 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:55.508488    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:00:56.055102 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:56.055070    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvgrv"
Jul 30 17:00:56.055546 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:56.055169    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hlj5"
Jul 30 17:00:56.055546 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:56.055170    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvgrv" podUID="fc0a3112-260e-47c8-87fd-bf3666ea17d0"
Jul 30 17:00:56.055546 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:56.055191    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-55hfh"
Jul 30 17:00:56.055546 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:56.055257    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9hlj5" podUID="51afa6bb-724f-47ab-b47b-36dc8a61ca25"
Jul 30 17:00:56.055546 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:56.055326    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-55hfh" podUID="e85f9d71-fe3d-49d8-93e2-e4ac101603c7"
Jul 30 17:00:56.401619 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:56.401540    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:00:56.495563 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:56.495534    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:00:56.703784 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:56.703703    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:00:56.904664 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:56.904624    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:00:57.101972 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:57.101942    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:00:57.274109 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:57.274073    2555 generic.go:356] "Generic (PLEG): container finished" podID="9fcdb3a5-74bc-44be-993f-fbc90e5db202" containerID="cefcc4c7a8f40f28b73b656dc1f0f3a20681167ae62a60a3d237432e9dcc9437" exitCode=0
Jul 30 17:00:57.274259 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:57.274125    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qbp2g" event={"ID":"9fcdb3a5-74bc-44be-993f-fbc90e5db202","Type":"ContainerDied","Data":"cefcc4c7a8f40f28b73b656dc1f0f3a20681167ae62a60a3d237432e9dcc9437"}
Jul 30 17:00:57.301729 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:57.301696    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:00:58.054479 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:58.054444    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-55hfh"
Jul 30 17:00:58.054673 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:58.054449    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvgrv"
Jul 30 17:00:58.054673 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:58.054584    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hlj5"
Jul 30 17:00:58.054673 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:58.054589    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-55hfh" podUID="e85f9d71-fe3d-49d8-93e2-e4ac101603c7"
Jul 30 17:00:58.054673 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:58.054655    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvgrv" podUID="fc0a3112-260e-47c8-87fd-bf3666ea17d0"
Jul 30 17:00:58.054889 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:58.054757    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9hlj5" podUID="51afa6bb-724f-47ab-b47b-36dc8a61ca25"
Jul 30 17:00:58.504481 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:58.504402    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:00:58.702566 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:58.702528    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:00:58.903369 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:58.903341    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:00:59.101429 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:59.101397    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:00:59.307051 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:59.306998    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:00:59.636818 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:59.636728    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmqln\" (UniqueName: \"kubernetes.io/projected/fc0a3112-260e-47c8-87fd-bf3666ea17d0-kube-api-access-rmqln\") pod \"network-check-target-qvgrv\" (UID: \"fc0a3112-260e-47c8-87fd-bf3666ea17d0\") " pod="openshift-network-diagnostics/network-check-target-qvgrv"
Jul 30 17:00:59.636818 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:59.636766    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/51afa6bb-724f-47ab-b47b-36dc8a61ca25-original-pull-secret\") pod \"global-pull-secret-syncer-9hlj5\" (UID: \"51afa6bb-724f-47ab-b47b-36dc8a61ca25\") " pod="kube-system/global-pull-secret-syncer-9hlj5"
Jul 30 17:00:59.636818 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:00:59.636803    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e85f9d71-fe3d-49d8-93e2-e4ac101603c7-metrics-certs\") pod \"network-metrics-daemon-55hfh\" (UID: \"e85f9d71-fe3d-49d8-93e2-e4ac101603c7\") " pod="openshift-multus/network-metrics-daemon-55hfh"
Jul 30 17:00:59.637402 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:59.636896    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 30 17:00:59.637402 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:59.636910    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 30 17:00:59.637402 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:59.636932    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 30 17:00:59.637402 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:59.636945    2555 projected.go:196] Error preparing data for projected volume kube-api-access-rmqln for pod openshift-network-diagnostics/network-check-target-qvgrv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 17:00:59.637402 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:59.636967    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51afa6bb-724f-47ab-b47b-36dc8a61ca25-original-pull-secret podName:51afa6bb-724f-47ab-b47b-36dc8a61ca25 nodeName:}" failed. No retries permitted until 2026-07-30 17:01:31.636945148 +0000 UTC m=+66.124312543 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/51afa6bb-724f-47ab-b47b-36dc8a61ca25-original-pull-secret") pod "global-pull-secret-syncer-9hlj5" (UID: "51afa6bb-724f-47ab-b47b-36dc8a61ca25") : object "kube-system"/"original-pull-secret" not registered
Jul 30 17:00:59.637402 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:59.636991    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fc0a3112-260e-47c8-87fd-bf3666ea17d0-kube-api-access-rmqln podName:fc0a3112-260e-47c8-87fd-bf3666ea17d0 nodeName:}" failed. No retries permitted until 2026-07-30 17:01:31.636978795 +0000 UTC m=+66.124346174 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-rmqln" (UniqueName: "kubernetes.io/projected/fc0a3112-260e-47c8-87fd-bf3666ea17d0-kube-api-access-rmqln") pod "network-check-target-qvgrv" (UID: "fc0a3112-260e-47c8-87fd-bf3666ea17d0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 17:00:59.637402 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:59.636910    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 17:00:59.637402 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:00:59.637044    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e85f9d71-fe3d-49d8-93e2-e4ac101603c7-metrics-certs podName:e85f9d71-fe3d-49d8-93e2-e4ac101603c7 nodeName:}" failed. No retries permitted until 2026-07-30 17:01:31.637033133 +0000 UTC m=+66.124400517 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e85f9d71-fe3d-49d8-93e2-e4ac101603c7-metrics-certs") pod "network-metrics-daemon-55hfh" (UID: "e85f9d71-fe3d-49d8-93e2-e4ac101603c7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 17:01:00.054945 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:00.054910    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvgrv"
Jul 30 17:01:00.055128 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:00.054949    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-55hfh"
Jul 30 17:01:00.055128 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:00.054914    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hlj5"
Jul 30 17:01:00.055128 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:01:00.055063    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvgrv" podUID="fc0a3112-260e-47c8-87fd-bf3666ea17d0"
Jul 30 17:01:00.055289 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:01:00.055170    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9hlj5" podUID="51afa6bb-724f-47ab-b47b-36dc8a61ca25"
Jul 30 17:01:00.055289 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:01:00.055267    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-55hfh" podUID="e85f9d71-fe3d-49d8-93e2-e4ac101603c7"
Jul 30 17:01:02.054653 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:02.054471    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvgrv"
Jul 30 17:01:02.055040 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:02.054471    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hlj5"
Jul 30 17:01:02.055040 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:01:02.054730    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvgrv" podUID="fc0a3112-260e-47c8-87fd-bf3666ea17d0"
Jul 30 17:01:02.055040 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:01:02.054789    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9hlj5" podUID="51afa6bb-724f-47ab-b47b-36dc8a61ca25"
Jul 30 17:01:02.055040 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:02.054488    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-55hfh"
Jul 30 17:01:02.055040 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:01:02.054855    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-55hfh" podUID="e85f9d71-fe3d-49d8-93e2-e4ac101603c7"
Jul 30 17:01:02.287623 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:02.287593    2555 generic.go:356] "Generic (PLEG): container finished" podID="9fcdb3a5-74bc-44be-993f-fbc90e5db202" containerID="d9fb92dceff4b3910fe7c466e556767747cfbb20aa37c07a3e7fb2ee40f55755" exitCode=0
Jul 30 17:01:02.287792 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:02.287650    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qbp2g" event={"ID":"9fcdb3a5-74bc-44be-993f-fbc90e5db202","Type":"ContainerDied","Data":"d9fb92dceff4b3910fe7c466e556767747cfbb20aa37c07a3e7fb2ee40f55755"}
Jul 30 17:01:03.295046 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:03.295001    2555 generic.go:356] "Generic (PLEG): container finished" podID="9fcdb3a5-74bc-44be-993f-fbc90e5db202" containerID="3157f2ad6f949c24b92f57a3ae6b11a653461b15dc0f6d128941ccd467175280" exitCode=0
Jul 30 17:01:03.295046 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:03.295039    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qbp2g" event={"ID":"9fcdb3a5-74bc-44be-993f-fbc90e5db202","Type":"ContainerDied","Data":"3157f2ad6f949c24b92f57a3ae6b11a653461b15dc0f6d128941ccd467175280"}
Jul 30 17:01:04.054648 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:04.054616    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvgrv"
Jul 30 17:01:04.054820 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:04.054616    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-55hfh"
Jul 30 17:01:04.054820 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:01:04.054718    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvgrv" podUID="fc0a3112-260e-47c8-87fd-bf3666ea17d0"
Jul 30 17:01:04.054820 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:01:04.054809    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-55hfh" podUID="e85f9d71-fe3d-49d8-93e2-e4ac101603c7"
Jul 30 17:01:04.054950 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:04.054619    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hlj5"
Jul 30 17:01:04.054950 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:01:04.054887    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9hlj5" podUID="51afa6bb-724f-47ab-b47b-36dc8a61ca25"
Jul 30 17:01:04.300247 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:04.300215    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qbp2g" event={"ID":"9fcdb3a5-74bc-44be-993f-fbc90e5db202","Type":"ContainerStarted","Data":"de63a4cd8782d14114a947422952e69601c4262803a5062f13d52e3a854fa63d"}
Jul 30 17:01:04.324446 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:04.324363    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-qbp2g" podStartSLOduration=13.311092276 podStartE2EDuration="38.324348106s" totalImagesPullingTime="25.01325583s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 17:00:26 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:01:04.324053925 +0000 UTC m=+38.811421323" watchObservedRunningTime="2026-07-30 17:01:04.324348106 +0000 UTC m=+38.811715506"
Jul 30 17:01:06.055991 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:06.055951    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvgrv"
Jul 30 17:01:06.056480 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:01:06.056070    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvgrv" podUID="fc0a3112-260e-47c8-87fd-bf3666ea17d0"
Jul 30 17:01:06.056480 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:06.056145    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-55hfh"
Jul 30 17:01:06.056480 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:01:06.056236    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-55hfh" podUID="e85f9d71-fe3d-49d8-93e2-e4ac101603c7"
Jul 30 17:01:06.056480 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:06.056283    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hlj5"
Jul 30 17:01:06.056480 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:01:06.056360    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9hlj5" podUID="51afa6bb-724f-47ab-b47b-36dc8a61ca25"
Jul 30 17:01:07.078342 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:07.078297    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:07.679317 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:07.679283    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:07.795302 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:07.795254    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:08.055217 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:08.055187    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-55hfh"
Jul 30 17:01:08.055217 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:08.055199    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvgrv"
Jul 30 17:01:08.055464 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:01:08.055293    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-55hfh" podUID="e85f9d71-fe3d-49d8-93e2-e4ac101603c7"
Jul 30 17:01:08.055464 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:01:08.055354    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvgrv" podUID="fc0a3112-260e-47c8-87fd-bf3666ea17d0"
Jul 30 17:01:08.055464 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:08.055394    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hlj5"
Jul 30 17:01:08.055558 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:01:08.055480    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9hlj5" podUID="51afa6bb-724f-47ab-b47b-36dc8a61ca25"
Jul 30 17:01:08.874198 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:08.874163    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9hlj5"]
Jul 30 17:01:08.874884 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:08.874302    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hlj5"
Jul 30 17:01:08.874884 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:01:08.874398    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9hlj5" podUID="51afa6bb-724f-47ab-b47b-36dc8a61ca25"
Jul 30 17:01:08.876623 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:08.876595    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qvgrv"]
Jul 30 17:01:08.876890 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:08.876875    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvgrv"
Jul 30 17:01:08.877057 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:01:08.877038    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvgrv" podUID="fc0a3112-260e-47c8-87fd-bf3666ea17d0"
Jul 30 17:01:08.877823 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:08.877475    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-55hfh"]
Jul 30 17:01:08.877823 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:08.877566    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-55hfh"
Jul 30 17:01:08.877823 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:01:08.877665    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-55hfh" podUID="e85f9d71-fe3d-49d8-93e2-e4ac101603c7"
Jul 30 17:01:09.877078 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:09.877044    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:09.930900 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:09.930873    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:10.054571 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:10.054536    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvgrv"
Jul 30 17:01:10.054750 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:01:10.054655    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvgrv" podUID="fc0a3112-260e-47c8-87fd-bf3666ea17d0"
Jul 30 17:01:11.055259 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:11.055227    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-55hfh"
Jul 30 17:01:11.055720 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:11.055227    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hlj5"
Jul 30 17:01:11.055720 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:01:11.055337    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-55hfh" podUID="e85f9d71-fe3d-49d8-93e2-e4ac101603c7"
Jul 30 17:01:11.055720 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:01:11.055402    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9hlj5" podUID="51afa6bb-724f-47ab-b47b-36dc8a61ca25"
Jul 30 17:01:11.969928 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:11.969891    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:12.055098 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:12.055066    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvgrv"
Jul 30 17:01:12.055261 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:01:12.055177    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvgrv" podUID="fc0a3112-260e-47c8-87fd-bf3666ea17d0"
Jul 30 17:01:12.069723 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:12.069696    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:12.483355 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:12.483325    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:12.583571 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:12.583543    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:12.639115 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:12.639083    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:13.055159 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:13.055127    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hlj5"
Jul 30 17:01:13.055332 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:13.055137    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-55hfh"
Jul 30 17:01:13.055332 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:01:13.055227    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9hlj5" podUID="51afa6bb-724f-47ab-b47b-36dc8a61ca25"
Jul 30 17:01:13.055332 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:01:13.055307    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-55hfh" podUID="e85f9d71-fe3d-49d8-93e2-e4ac101603c7"
Jul 30 17:01:13.229535 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:13.229347    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-190.ec2.internal" event="NodeReady"
Jul 30 17:01:13.229903 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:13.229639    2555 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 30 17:01:13.274184 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:13.274153    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-g5pvs"]
Jul 30 17:01:13.310484 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:13.310376    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-tkbtl"]
Jul 30 17:01:13.310625 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:13.310532    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-g5pvs"
Jul 30 17:01:13.313105 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:13.313084    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 30 17:01:13.313251 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:13.313085    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-c76qr\""
Jul 30 17:01:13.313251 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:13.313150    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 30 17:01:13.313251 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:13.313163    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 30 17:01:13.325621 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:13.325596    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-g5pvs"]
Jul 30 17:01:13.325621 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:13.325627    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tkbtl"]
Jul 30 17:01:13.325790 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:13.325660    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-l9zx6"]
Jul 30 17:01:13.326078 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:13.326052    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tkbtl"
Jul 30 17:01:13.329759 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:13.329724    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 30 17:01:13.329889 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:13.329734    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 30 17:01:13.329889 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:13.329782    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 30 17:01:13.329973 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:13.329884    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-k8vp7\""
Jul 30 17:01:13.330195 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:13.330174    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 30 17:01:13.350116 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:13.350088    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-l9zx6"]
Jul 30 17:01:13.350270 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:13.350196    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-l9zx6"
Jul 30 17:01:13.352761 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:13.352730    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 30 17:01:13.352761 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:13.352754    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hvt8c\""
Jul 30 17:01:13.352935 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:13.352737    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 30 17:01:13.420195 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:13.420161    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:13.438190 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:13.438163    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a340f4e6-daaf-450e-b880-a3658b1296ef-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tkbtl\" (UID: \"a340f4e6-daaf-450e-b880-a3658b1296ef\") " pod="openshift-insights/insights-runtime-extractor-tkbtl"
Jul 30 17:01:13.438289 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:13.438196    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwsnc\" (UniqueName: \"kubernetes.io/projected/a340f4e6-daaf-450e-b880-a3658b1296ef-kube-api-access-mwsnc\") pod \"insights-runtime-extractor-tkbtl\" (UID: \"a340f4e6-daaf-450e-b880-a3658b1296ef\") " pod="openshift-insights/insights-runtime-extractor-tkbtl"
Jul 30 17:01:13.438289 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:13.438213    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqbc8\" (UniqueName: \"kubernetes.io/projected/ac14b3bf-789c-4419-a46b-55fab58dcfd8-kube-api-access-pqbc8\") pod \"ingress-canary-g5pvs\" (UID: \"ac14b3bf-789c-4419-a46b-55fab58dcfd8\") " pod="openshift-ingress-canary/ingress-canary-g5pvs"
Jul 30 17:01:13.438289 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:13.438233    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/13abbab1-dfcf-4102-9a67-54f9b636e1f3-metrics-tls\") pod \"dns-default-l9zx6\" (UID: \"13abbab1-dfcf-4102-9a67-54f9b636e1f3\") " pod="openshift-dns/dns-default-l9zx6"
Jul 30 17:01:13.438289 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:13.438276    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13abbab1-dfcf-4102-9a67-54f9b636e1f3-tmp-dir\") pod \"dns-default-l9zx6\" (UID: \"13abbab1-dfcf-4102-9a67-54f9b636e1f3\") " pod="openshift-dns/dns-default-l9zx6"
Jul 30 17:01:13.438434 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:13.438293    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a340f4e6-daaf-450e-b880-a3658b1296ef-crio-socket\") pod \"insights-runtime-extractor-tkbtl\" (UID: \"a340f4e6-daaf-450e-b880-a3658b1296ef\") " pod="openshift-insights/insights-runtime-extractor-tkbtl"
Jul 30 17:01:13.438434 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:13.438314    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a340f4e6-daaf-450e-b880-a3658b1296ef-data-volume\") pod \"insights-runtime-extractor-tkbtl\" (UID: \"a340f4e6-daaf-450e-b880-a3658b1296ef\") " pod="openshift-insights/insights-runtime-extractor-tkbtl"
Jul 30 17:01:13.438434 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:13.438354    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ac14b3bf-789c-4419-a46b-55fab58dcfd8-cert\") pod \"ingress-canary-g5pvs\" (UID: \"ac14b3bf-789c-4419-a46b-55fab58dcfd8\") " pod="openshift-ingress-canary/ingress-canary-g5pvs"
Jul 30 17:01:13.438434 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:13.438374    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5gks\" (UniqueName: \"kubernetes.io/projected/13abbab1-dfcf-4102-9a67-54f9b636e1f3-kube-api-access-m5gks\") pod \"dns-default-l9zx6\" (UID: \"13abbab1-dfcf-4102-9a67-54f9b636e1f3\") " pod="openshift-dns/dns-default-l9zx6"
Jul 30 17:01:13.438434 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:13.438398    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/13abbab1-dfcf-4102-9a67-54f9b636e1f3-config-volume\") pod \"dns-default-l9zx6\" (UID: \"13abbab1-dfcf-4102-9a67-54f9b636e1f3\") " pod="openshift-dns/dns-default-l9zx6"
Jul 30 17:01:13.438434 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:13.438432    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a340f4e6-daaf-450e-b880-a3658b1296ef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tkbtl\" (UID: \"a340f4e6-daaf-450e-b880-a3658b1296ef\") " pod="openshift-insights/insights-runtime-extractor-tkbtl"
Jul 30 17:01:13.484733 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:13.484709    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:13.539431 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:13.539399    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a340f4e6-daaf-450e-b880-a3658b1296ef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tkbtl\" (UID: \"a340f4e6-daaf-450e-b880-a3658b1296ef\") " pod="openshift-insights/insights-runtime-extractor-tkbtl"
Jul 30 17:01:13.539588 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:13.539444    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a340f4e6-daaf-450e-b880-a3658b1296ef-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tkbtl\" (UID: \"a340f4e6-daaf-450e-b880-a3658b1296ef\") " pod="openshift-insights/insights-runtime-extractor-tkbtl"
Jul 30 17:01:13.539588 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:13.539559    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwsnc\" (UniqueName: \"kubernetes.io/projected/a340f4e6-daaf-450e-b880-a3658b1296ef-kube-api-access-mwsnc\") pod \"insights-runtime-extractor-tkbtl\" (UID: \"a340f4e6-daaf-450e-b880-a3658b1296ef\") " pod="openshift-insights/insights-runtime-extractor-tkbtl"
Jul 30 17:01:13.539680 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:13.539590    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqbc8\" (UniqueName: \"kubernetes.io/projected/ac14b3bf-789c-4419-a46b-55fab58dcfd8-kube-api-access-pqbc8\") pod \"ingress-canary-g5pvs\" (UID: \"ac14b3bf-789c-4419-a46b-55fab58dcfd8\") " pod="openshift-ingress-canary/ingress-canary-g5pvs"
Jul 30 17:01:13.539680 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:13.539619    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/13abbab1-dfcf-4102-9a67-54f9b636e1f3-metrics-tls\") pod \"dns-default-l9zx6\" (UID: \"13abbab1-dfcf-4102-9a67-54f9b636e1f3\") " pod="openshift-dns/dns-default-l9zx6"
Jul 30 17:01:13.539680 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:13.539667    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13abbab1-dfcf-4102-9a67-54f9b636e1f3-tmp-dir\") pod \"dns-default-l9zx6\" (UID: \"13abbab1-dfcf-4102-9a67-54f9b636e1f3\") " pod="openshift-dns/dns-default-l9zx6"
Jul 30 17:01:13.539806 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:13.539697    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a340f4e6-daaf-450e-b880-a3658b1296ef-crio-socket\") pod \"insights-runtime-extractor-tkbtl\" (UID: \"a340f4e6-daaf-450e-b880-a3658b1296ef\") " pod="openshift-insights/insights-runtime-extractor-tkbtl"
Jul 30 17:01:13.539806 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:13.539720    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a340f4e6-daaf-450e-b880-a3658b1296ef-data-volume\") pod \"insights-runtime-extractor-tkbtl\" (UID: \"a340f4e6-daaf-450e-b880-a3658b1296ef\") " pod="openshift-insights/insights-runtime-extractor-tkbtl"
Jul 30 17:01:13.539806 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:13.539744    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ac14b3bf-789c-4419-a46b-55fab58dcfd8-cert\") pod \"ingress-canary-g5pvs\" (UID: \"ac14b3bf-789c-4419-a46b-55fab58dcfd8\") " pod="openshift-ingress-canary/ingress-canary-g5pvs"
Jul 30 17:01:13.539806 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:13.539763    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5gks\" (UniqueName: \"kubernetes.io/projected/13abbab1-dfcf-4102-9a67-54f9b636e1f3-kube-api-access-m5gks\") pod \"dns-default-l9zx6\" (UID: \"13abbab1-dfcf-4102-9a67-54f9b636e1f3\") " pod="openshift-dns/dns-default-l9zx6"
Jul 30 17:01:13.539806 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:13.539784    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/13abbab1-dfcf-4102-9a67-54f9b636e1f3-config-volume\") pod \"dns-default-l9zx6\" (UID: \"13abbab1-dfcf-4102-9a67-54f9b636e1f3\") " pod="openshift-dns/dns-default-l9zx6"
Jul 30 17:01:13.540033 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:13.539965    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a340f4e6-daaf-450e-b880-a3658b1296ef-crio-socket\") pod \"insights-runtime-extractor-tkbtl\" (UID: \"a340f4e6-daaf-450e-b880-a3658b1296ef\") " pod="openshift-insights/insights-runtime-extractor-tkbtl"
Jul 30 17:01:13.540135 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:13.540112    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13abbab1-dfcf-4102-9a67-54f9b636e1f3-tmp-dir\") pod \"dns-default-l9zx6\" (UID: \"13abbab1-dfcf-4102-9a67-54f9b636e1f3\") " pod="openshift-dns/dns-default-l9zx6"
Jul 30 17:01:13.540318 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:13.540298    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a340f4e6-daaf-450e-b880-a3658b1296ef-data-volume\") pod \"insights-runtime-extractor-tkbtl\" (UID: \"a340f4e6-daaf-450e-b880-a3658b1296ef\") " pod="openshift-insights/insights-runtime-extractor-tkbtl"
Jul 30 17:01:13.540496 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:13.540473    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a340f4e6-daaf-450e-b880-a3658b1296ef-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tkbtl\" (UID: \"a340f4e6-daaf-450e-b880-a3658b1296ef\") " pod="openshift-insights/insights-runtime-extractor-tkbtl"
Jul 30 17:01:13.540541 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:13.540473    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/13abbab1-dfcf-4102-9a67-54f9b636e1f3-config-volume\") pod \"dns-default-l9zx6\" (UID: \"13abbab1-dfcf-4102-9a67-54f9b636e1f3\") " pod="openshift-dns/dns-default-l9zx6"
Jul 30 17:01:13.543432 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:13.543413    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a340f4e6-daaf-450e-b880-a3658b1296ef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tkbtl\" (UID: \"a340f4e6-daaf-450e-b880-a3658b1296ef\") " pod="openshift-insights/insights-runtime-extractor-tkbtl"
Jul 30 17:01:13.543559 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:13.543455    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/13abbab1-dfcf-4102-9a67-54f9b636e1f3-metrics-tls\") pod \"dns-default-l9zx6\" (UID: \"13abbab1-dfcf-4102-9a67-54f9b636e1f3\") " pod="openshift-dns/dns-default-l9zx6"
Jul 30 17:01:13.543559 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:13.543549    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ac14b3bf-789c-4419-a46b-55fab58dcfd8-cert\") pod \"ingress-canary-g5pvs\" (UID: \"ac14b3bf-789c-4419-a46b-55fab58dcfd8\") " pod="openshift-ingress-canary/ingress-canary-g5pvs"
Jul 30 17:01:13.549445 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:13.549418    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5gks\" (UniqueName: \"kubernetes.io/projected/13abbab1-dfcf-4102-9a67-54f9b636e1f3-kube-api-access-m5gks\") pod \"dns-default-l9zx6\" (UID: \"13abbab1-dfcf-4102-9a67-54f9b636e1f3\") " pod="openshift-dns/dns-default-l9zx6"
Jul 30 17:01:13.549558 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:13.549545    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqbc8\" (UniqueName: \"kubernetes.io/projected/ac14b3bf-789c-4419-a46b-55fab58dcfd8-kube-api-access-pqbc8\") pod \"ingress-canary-g5pvs\" (UID: \"ac14b3bf-789c-4419-a46b-55fab58dcfd8\") " pod="openshift-ingress-canary/ingress-canary-g5pvs"
Jul 30 17:01:13.549616 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:13.549567    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwsnc\" (UniqueName: \"kubernetes.io/projected/a340f4e6-daaf-450e-b880-a3658b1296ef-kube-api-access-mwsnc\") pod \"insights-runtime-extractor-tkbtl\" (UID: \"a340f4e6-daaf-450e-b880-a3658b1296ef\") " pod="openshift-insights/insights-runtime-extractor-tkbtl"
Jul 30 17:01:13.620639 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:13.620548    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-g5pvs"
Jul 30 17:01:13.635756 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:13.635723    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tkbtl"
Jul 30 17:01:13.660173 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:13.660147    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-l9zx6"
Jul 30 17:01:13.843950 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:13.843921    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-g5pvs"]
Jul 30 17:01:13.847697 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:13.847672    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tkbtl"]
Jul 30 17:01:13.847864 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:01:13.847844    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac14b3bf_789c_4419_a46b_55fab58dcfd8.slice/crio-c84cbf99eb3f7d4db4c4fe9d89ea46e6123900ba106389a2bb383d97d61256b7 WatchSource:0}: Error finding container c84cbf99eb3f7d4db4c4fe9d89ea46e6123900ba106389a2bb383d97d61256b7: Status 404 returned error can't find the container with id c84cbf99eb3f7d4db4c4fe9d89ea46e6123900ba106389a2bb383d97d61256b7
Jul 30 17:01:13.851701 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:13.851677    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-l9zx6"]
Jul 30 17:01:13.853463 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:01:13.853439    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda340f4e6_daaf_450e_b880_a3658b1296ef.slice/crio-27d87a5e41ab82aef2c0b79fa2f65c16dac8b706eeaaaa890551944f007832f2 WatchSource:0}: Error finding container 27d87a5e41ab82aef2c0b79fa2f65c16dac8b706eeaaaa890551944f007832f2: Status 404 returned error can't find the container with id 27d87a5e41ab82aef2c0b79fa2f65c16dac8b706eeaaaa890551944f007832f2
Jul 30 17:01:13.854861 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:01:13.854834    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13abbab1_dfcf_4102_9a67_54f9b636e1f3.slice/crio-de951df5e6c3c10d4357cee312dad88a8041b4a5d86867615653ebf9125b2218 WatchSource:0}: Error finding container de951df5e6c3c10d4357cee312dad88a8041b4a5d86867615653ebf9125b2218: Status 404 returned error can't find the container with id de951df5e6c3c10d4357cee312dad88a8041b4a5d86867615653ebf9125b2218
Jul 30 17:01:14.058097 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:14.058070    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvgrv"
Jul 30 17:01:14.060809 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:14.060786    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 30 17:01:14.060945 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:14.060809    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 30 17:01:14.060945 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:14.060790    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wcszl\""
Jul 30 17:01:14.323670 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:14.323590    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-l9zx6" event={"ID":"13abbab1-dfcf-4102-9a67-54f9b636e1f3","Type":"ContainerStarted","Data":"de951df5e6c3c10d4357cee312dad88a8041b4a5d86867615653ebf9125b2218"}
Jul 30 17:01:14.324972 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:14.324947    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tkbtl" event={"ID":"a340f4e6-daaf-450e-b880-a3658b1296ef","Type":"ContainerStarted","Data":"27d87a5e41ab82aef2c0b79fa2f65c16dac8b706eeaaaa890551944f007832f2"}
Jul 30 17:01:14.326152 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:14.326115    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-g5pvs" event={"ID":"ac14b3bf-789c-4419-a46b-55fab58dcfd8","Type":"ContainerStarted","Data":"c84cbf99eb3f7d4db4c4fe9d89ea46e6123900ba106389a2bb383d97d61256b7"}
Jul 30 17:01:15.055149 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:15.054899    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hlj5"
Jul 30 17:01:15.055330 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:15.054899    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-55hfh"
Jul 30 17:01:15.060599 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:15.060574    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4wpnm\""
Jul 30 17:01:15.060862 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:15.060846    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 30 17:01:15.061093 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:15.061073    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 30 17:01:19.053758 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:19.053571    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:19.185468 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:19.185432    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:19.265909 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:19.265864    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:19.511336 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:19.511291    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:19.803340 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:19.803305    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:19.858085 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:19.858049    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:20.346080 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:20.346044    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-l9zx6" event={"ID":"13abbab1-dfcf-4102-9a67-54f9b636e1f3","Type":"ContainerStarted","Data":"b568129e585e4ba51479f118d18d793ff965f73fafe9d68f13e3c10597ecfdde"}
Jul 30 17:01:20.346807 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:20.346157    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-l9zx6" event={"ID":"13abbab1-dfcf-4102-9a67-54f9b636e1f3","Type":"ContainerStarted","Data":"9c88751e6eb66d094e1b8c92ce7f2aec9b762276b43c80cd676713d527c9ab9e"}
Jul 30 17:01:20.348155 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:20.348127    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tkbtl" event={"ID":"a340f4e6-daaf-450e-b880-a3658b1296ef","Type":"ContainerStarted","Data":"951cbe73ae8f1cb239e9ca2360c64c50740cd4093a30fc876e9746ceb94abf8e"}
Jul 30 17:01:20.349564 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:20.349540    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-g5pvs" event={"ID":"ac14b3bf-789c-4419-a46b-55fab58dcfd8","Type":"ContainerStarted","Data":"1dafb8549c526853d2ecfd8573a97bd4e0a14561fa9e4da35026097c03a1aa22"}
Jul 30 17:01:20.365425 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:20.365285    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-g5pvs" podStartSLOduration=1.181745877 podStartE2EDuration="7.365264977s" totalImagesPullingTime="6.1835191s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:01:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:01:20.364841103 +0000 UTC m=+54.852208505" watchObservedRunningTime="2026-07-30 17:01:20.365264977 +0000 UTC m=+54.852632379"
Jul 30 17:01:21.353599 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:21.353374    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-l9zx6"
Jul 30 17:01:21.372933 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:21.372713    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-l9zx6" podStartSLOduration=2.197642574 podStartE2EDuration="8.372691124s" totalImagesPullingTime="6.17504855s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:01:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:01:21.37226986 +0000 UTC m=+55.859637262" watchObservedRunningTime="2026-07-30 17:01:21.372691124 +0000 UTC m=+55.860058526"
Jul 30 17:01:21.628807 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:21.628716    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:22.019861 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:22.019788    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-lgsc2"]
Jul 30 17:01:22.055976 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:22.055944    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-lgsc2"]
Jul 30 17:01:22.056156 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:22.056041    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-lgsc2"
Jul 30 17:01:22.058761 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:22.058732    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 30 17:01:22.058761 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:22.058752    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-c9jch\""
Jul 30 17:01:22.058987 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:22.058745    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 30 17:01:22.112673 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:22.112634    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrqp9\" (UniqueName: \"kubernetes.io/projected/7bb3cf69-d970-4823-b2f4-f9f726efaf9d-kube-api-access-wrqp9\") pod \"downloads-7c97dc4555-lgsc2\" (UID: \"7bb3cf69-d970-4823-b2f4-f9f726efaf9d\") " pod="openshift-console/downloads-7c97dc4555-lgsc2"
Jul 30 17:01:22.213835 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:22.213797    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrqp9\" (UniqueName: \"kubernetes.io/projected/7bb3cf69-d970-4823-b2f4-f9f726efaf9d-kube-api-access-wrqp9\") pod \"downloads-7c97dc4555-lgsc2\" (UID: \"7bb3cf69-d970-4823-b2f4-f9f726efaf9d\") " pod="openshift-console/downloads-7c97dc4555-lgsc2"
Jul 30 17:01:22.224908 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:22.224881    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrqp9\" (UniqueName: \"kubernetes.io/projected/7bb3cf69-d970-4823-b2f4-f9f726efaf9d-kube-api-access-wrqp9\") pod \"downloads-7c97dc4555-lgsc2\" (UID: \"7bb3cf69-d970-4823-b2f4-f9f726efaf9d\") " pod="openshift-console/downloads-7c97dc4555-lgsc2"
Jul 30 17:01:22.367541 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:22.367509    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-lgsc2"
Jul 30 17:01:22.584312 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:22.584279    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:22.641802 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:22.641728    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:22.699826 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:22.699791    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:23.106763 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:23.106729    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:23.161670 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:23.161642    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:23.302047 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:23.302001    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:23.403315 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:23.403246    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:23.706362 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:23.706285    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:23.917912 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:23.917880    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:24.149691 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:24.149654    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:24.290110 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:24.290081    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-lgsc2"]
Jul 30 17:01:24.293303 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:01:24.293277    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bb3cf69_d970_4823_b2f4_f9f726efaf9d.slice/crio-4ea01f84132f10920daf1ea9170f76edaecd9b4fdd89b0359a42b6219b10c95a WatchSource:0}: Error finding container 4ea01f84132f10920daf1ea9170f76edaecd9b4fdd89b0359a42b6219b10c95a: Status 404 returned error can't find the container with id 4ea01f84132f10920daf1ea9170f76edaecd9b4fdd89b0359a42b6219b10c95a
Jul 30 17:01:24.296999 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:24.296979    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:24.337953 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:24.337921    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-7s9bw"]
Jul 30 17:01:24.339827 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:24.339811    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-7s9bw"
Jul 30 17:01:24.342188 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:24.342165    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 30 17:01:24.342374 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:24.342355    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 30 17:01:24.342446 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:24.342410    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 30 17:01:24.342501 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:24.342458    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-x8df2\""
Jul 30 17:01:24.342579 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:24.342559    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 30 17:01:24.342682 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:24.342637    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 30 17:01:24.349738 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:24.349720    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-7s9bw"]
Jul 30 17:01:24.361451 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:24.361424    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tkbtl" event={"ID":"a340f4e6-daaf-450e-b880-a3658b1296ef","Type":"ContainerStarted","Data":"243c4adb2e451501d972aba18d1aeee9263c69844c466af081c4c6022b826bcb"}
Jul 30 17:01:24.362443 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:24.362421    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-lgsc2" event={"ID":"7bb3cf69-d970-4823-b2f4-f9f726efaf9d","Type":"ContainerStarted","Data":"4ea01f84132f10920daf1ea9170f76edaecd9b4fdd89b0359a42b6219b10c95a"}
Jul 30 17:01:24.387604 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:24.387579    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:24.433873 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:24.433798    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/14057a67-fd8e-40e9-91c9-3ff562752bbf-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-7s9bw\" (UID: \"14057a67-fd8e-40e9-91c9-3ff562752bbf\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-7s9bw"
Jul 30 17:01:24.433873 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:24.433831    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbfqp\" (UniqueName: \"kubernetes.io/projected/14057a67-fd8e-40e9-91c9-3ff562752bbf-kube-api-access-dbfqp\") pod \"prometheus-operator-84d69f794d-7s9bw\" (UID: \"14057a67-fd8e-40e9-91c9-3ff562752bbf\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-7s9bw"
Jul 30 17:01:24.433873 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:24.433862    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14057a67-fd8e-40e9-91c9-3ff562752bbf-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-7s9bw\" (UID: \"14057a67-fd8e-40e9-91c9-3ff562752bbf\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-7s9bw"
Jul 30 17:01:24.434310 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:24.433879    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/14057a67-fd8e-40e9-91c9-3ff562752bbf-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-7s9bw\" (UID: \"14057a67-fd8e-40e9-91c9-3ff562752bbf\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-7s9bw"
Jul 30 17:01:24.534306 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:24.534264    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/14057a67-fd8e-40e9-91c9-3ff562752bbf-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-7s9bw\" (UID: \"14057a67-fd8e-40e9-91c9-3ff562752bbf\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-7s9bw"
Jul 30 17:01:24.534463 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:24.534352    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/14057a67-fd8e-40e9-91c9-3ff562752bbf-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-7s9bw\" (UID: \"14057a67-fd8e-40e9-91c9-3ff562752bbf\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-7s9bw"
Jul 30 17:01:24.534463 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:24.534375    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbfqp\" (UniqueName: \"kubernetes.io/projected/14057a67-fd8e-40e9-91c9-3ff562752bbf-kube-api-access-dbfqp\") pod \"prometheus-operator-84d69f794d-7s9bw\" (UID: \"14057a67-fd8e-40e9-91c9-3ff562752bbf\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-7s9bw"
Jul 30 17:01:24.534463 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:24.534408    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14057a67-fd8e-40e9-91c9-3ff562752bbf-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-7s9bw\" (UID: \"14057a67-fd8e-40e9-91c9-3ff562752bbf\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-7s9bw"
Jul 30 17:01:24.534680 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:01:24.534523    2555 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 30 17:01:24.534680 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:01:24.534607    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14057a67-fd8e-40e9-91c9-3ff562752bbf-prometheus-operator-tls podName:14057a67-fd8e-40e9-91c9-3ff562752bbf nodeName:}" failed. No retries permitted until 2026-07-30 17:01:25.034585382 +0000 UTC m=+59.521952768 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/14057a67-fd8e-40e9-91c9-3ff562752bbf-prometheus-operator-tls") pod "prometheus-operator-84d69f794d-7s9bw" (UID: "14057a67-fd8e-40e9-91c9-3ff562752bbf") : secret "prometheus-operator-tls" not found
Jul 30 17:01:24.535187 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:24.535170    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14057a67-fd8e-40e9-91c9-3ff562752bbf-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-7s9bw\" (UID: \"14057a67-fd8e-40e9-91c9-3ff562752bbf\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-7s9bw"
Jul 30 17:01:24.536919 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:24.536897    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/14057a67-fd8e-40e9-91c9-3ff562752bbf-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-7s9bw\" (UID: \"14057a67-fd8e-40e9-91c9-3ff562752bbf\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-7s9bw"
Jul 30 17:01:24.547161 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:24.547126    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbfqp\" (UniqueName: \"kubernetes.io/projected/14057a67-fd8e-40e9-91c9-3ff562752bbf-kube-api-access-dbfqp\") pod \"prometheus-operator-84d69f794d-7s9bw\" (UID: \"14057a67-fd8e-40e9-91c9-3ff562752bbf\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-7s9bw"
Jul 30 17:01:24.698365 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:24.698246    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:25.038836 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:25.038747    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/14057a67-fd8e-40e9-91c9-3ff562752bbf-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-7s9bw\" (UID: \"14057a67-fd8e-40e9-91c9-3ff562752bbf\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-7s9bw"
Jul 30 17:01:25.041525 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:25.041501    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/14057a67-fd8e-40e9-91c9-3ff562752bbf-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-7s9bw\" (UID: \"14057a67-fd8e-40e9-91c9-3ff562752bbf\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-7s9bw"
Jul 30 17:01:25.248908 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:25.248873    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-7s9bw"
Jul 30 17:01:25.340908 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:25.339957    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:25.399477 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:25.399394    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-7s9bw"]
Jul 30 17:01:25.402592 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:01:25.402558    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14057a67_fd8e_40e9_91c9_3ff562752bbf.slice/crio-eacef921ef87ebab2882da23820c4e2516b65a9473a79d1866de06f32b548622 WatchSource:0}: Error finding container eacef921ef87ebab2882da23820c4e2516b65a9473a79d1866de06f32b548622: Status 404 returned error can't find the container with id eacef921ef87ebab2882da23820c4e2516b65a9473a79d1866de06f32b548622
Jul 30 17:01:25.403102 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:25.403081    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:26.283386 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:26.283356    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-t87pw"
Jul 30 17:01:26.371290 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:26.371253    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-7s9bw" event={"ID":"14057a67-fd8e-40e9-91c9-3ff562752bbf","Type":"ContainerStarted","Data":"eacef921ef87ebab2882da23820c4e2516b65a9473a79d1866de06f32b548622"}
Jul 30 17:01:26.439175 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:26.439131    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:26.493776 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:26.493735    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:29.063862 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:29.063821    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:29.381994 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:29.381902    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-7s9bw" event={"ID":"14057a67-fd8e-40e9-91c9-3ff562752bbf","Type":"ContainerStarted","Data":"b4f7841e3c7d24e176309d38c50b4bbea02821d589632c4261c17f364d36a8c1"}
Jul 30 17:01:29.382174 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:29.382034    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-7s9bw" event={"ID":"14057a67-fd8e-40e9-91c9-3ff562752bbf","Type":"ContainerStarted","Data":"5fa8100f1e33457e195bf9534d6603506b6a153c4db35e4010ff603a7b96eaef"}
Jul 30 17:01:29.383862 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:29.383836    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tkbtl" event={"ID":"a340f4e6-daaf-450e-b880-a3658b1296ef","Type":"ContainerStarted","Data":"8da6c7abb12295b3f7abd64577c4b953db2018a50fa5597172cee07d304263a5"}
Jul 30 17:01:29.400504 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:29.400448    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-7s9bw" podStartSLOduration=2.013815614 podStartE2EDuration="5.400433925s" totalImagesPullingTime="3.386618311s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:01:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:01:29.399191292 +0000 UTC m=+63.886558694" watchObservedRunningTime="2026-07-30 17:01:29.400433925 +0000 UTC m=+63.887801327"
Jul 30 17:01:29.420880 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:29.420820    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-tkbtl" podStartSLOduration=1.677213721 podStartE2EDuration="16.420801719s" totalImagesPullingTime="14.743587998s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 17:01:13 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:01:29.419930739 +0000 UTC m=+63.907298139" watchObservedRunningTime="2026-07-30 17:01:29.420801719 +0000 UTC m=+63.908169098"
Jul 30 17:01:31.358064 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:31.358030    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-l9zx6"
Jul 30 17:01:31.463440 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:31.463413    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:31.656500 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:31.656412    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:31.691347 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:31.691308    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e85f9d71-fe3d-49d8-93e2-e4ac101603c7-metrics-certs\") pod \"network-metrics-daemon-55hfh\" (UID: \"e85f9d71-fe3d-49d8-93e2-e4ac101603c7\") " pod="openshift-multus/network-metrics-daemon-55hfh"
Jul 30 17:01:31.691535 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:31.691383    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmqln\" (UniqueName: \"kubernetes.io/projected/fc0a3112-260e-47c8-87fd-bf3666ea17d0-kube-api-access-rmqln\") pod \"network-check-target-qvgrv\" (UID: \"fc0a3112-260e-47c8-87fd-bf3666ea17d0\") " pod="openshift-network-diagnostics/network-check-target-qvgrv"
Jul 30 17:01:31.691535 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:31.691414    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/51afa6bb-724f-47ab-b47b-36dc8a61ca25-original-pull-secret\") pod \"global-pull-secret-syncer-9hlj5\" (UID: \"51afa6bb-724f-47ab-b47b-36dc8a61ca25\") " pod="kube-system/global-pull-secret-syncer-9hlj5"
Jul 30 17:01:31.693669 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:31.693648    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 30 17:01:31.693781 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:31.693684    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 30 17:01:31.693879 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:31.693843    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 30 17:01:31.704163 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:31.704139    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 30 17:01:31.704355 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:31.704333    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e85f9d71-fe3d-49d8-93e2-e4ac101603c7-metrics-certs\") pod \"network-metrics-daemon-55hfh\" (UID: \"e85f9d71-fe3d-49d8-93e2-e4ac101603c7\") " pod="openshift-multus/network-metrics-daemon-55hfh"
Jul 30 17:01:31.704425 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:31.704356    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/51afa6bb-724f-47ab-b47b-36dc8a61ca25-original-pull-secret\") pod \"global-pull-secret-syncer-9hlj5\" (UID: \"51afa6bb-724f-47ab-b47b-36dc8a61ca25\") " pod="kube-system/global-pull-secret-syncer-9hlj5"
Jul 30 17:01:31.714683 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:31.714656    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmqln\" (UniqueName: \"kubernetes.io/projected/fc0a3112-260e-47c8-87fd-bf3666ea17d0-kube-api-access-rmqln\") pod \"network-check-target-qvgrv\" (UID: \"fc0a3112-260e-47c8-87fd-bf3666ea17d0\") " pod="openshift-network-diagnostics/network-check-target-qvgrv"
Jul 30 17:01:31.717125 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:31.717102    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-qvrdk"]
Jul 30 17:01:31.721424 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:31.721401    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-qvrdk"
Jul 30 17:01:31.723855 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:31.723834    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 30 17:01:31.723959 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:31.723868    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 30 17:01:31.724215 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:31.724194    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-xbgk4\""
Jul 30 17:01:31.732737 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:31.732667    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-qvrdk"]
Jul 30 17:01:31.744179 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:31.744154    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-9q5tr"]
Jul 30 17:01:31.746152 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:31.746134    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9q5tr"
Jul 30 17:01:31.748648 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:31.748626    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 30 17:01:31.748765 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:31.748721    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 30 17:01:31.748839 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:31.748828    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-hq64z\""
Jul 30 17:01:31.748915 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:31.748897    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 30 17:01:31.768991 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:31.768972    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wcszl\""
Jul 30 17:01:31.777063 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:31.776839    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvgrv"
Jul 30 17:01:31.791869 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:31.791846    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45cb4db9-0f93-4fcb-80c5-20ff7c527aab-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-qvrdk\" (UID: \"45cb4db9-0f93-4fcb-80c5-20ff7c527aab\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-qvrdk"
Jul 30 17:01:31.791989 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:31.791876    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cpbg\" (UniqueName: \"kubernetes.io/projected/45cb4db9-0f93-4fcb-80c5-20ff7c527aab-kube-api-access-2cpbg\") pod \"openshift-state-metrics-b8f49f5c6-qvrdk\" (UID: \"45cb4db9-0f93-4fcb-80c5-20ff7c527aab\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-qvrdk"
Jul 30 17:01:31.791989 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:31.791900    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ed87dbe4-de48-4ff6-bb84-cac7b1719661-node-exporter-tls\") pod \"node-exporter-9q5tr\" (UID: \"ed87dbe4-de48-4ff6-bb84-cac7b1719661\") " pod="openshift-monitoring/node-exporter-9q5tr"
Jul 30 17:01:31.791989 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:31.791928    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/45cb4db9-0f93-4fcb-80c5-20ff7c527aab-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-qvrdk\" (UID: \"45cb4db9-0f93-4fcb-80c5-20ff7c527aab\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-qvrdk"
Jul 30 17:01:31.791989 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:31.791957    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ed87dbe4-de48-4ff6-bb84-cac7b1719661-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9q5tr\" (UID: \"ed87dbe4-de48-4ff6-bb84-cac7b1719661\") " pod="openshift-monitoring/node-exporter-9q5tr"
Jul 30 17:01:31.792221 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:31.791996    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ed87dbe4-de48-4ff6-bb84-cac7b1719661-root\") pod \"node-exporter-9q5tr\" (UID: \"ed87dbe4-de48-4ff6-bb84-cac7b1719661\") " pod="openshift-monitoring/node-exporter-9q5tr"
Jul 30 17:01:31.792221 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:31.792053    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ed87dbe4-de48-4ff6-bb84-cac7b1719661-node-exporter-accelerators-collector-config\") pod \"node-exporter-9q5tr\" (UID: \"ed87dbe4-de48-4ff6-bb84-cac7b1719661\") " pod="openshift-monitoring/node-exporter-9q5tr"
Jul 30 17:01:31.792221 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:31.792082    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/45cb4db9-0f93-4fcb-80c5-20ff7c527aab-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-qvrdk\" (UID: \"45cb4db9-0f93-4fcb-80c5-20ff7c527aab\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-qvrdk"
Jul 30 17:01:31.792221 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:31.792104    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ed87dbe4-de48-4ff6-bb84-cac7b1719661-node-exporter-wtmp\") pod \"node-exporter-9q5tr\" (UID: \"ed87dbe4-de48-4ff6-bb84-cac7b1719661\") " pod="openshift-monitoring/node-exporter-9q5tr"
Jul 30 17:01:31.792221 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:31.792135    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ed87dbe4-de48-4ff6-bb84-cac7b1719661-sys\") pod \"node-exporter-9q5tr\" (UID: \"ed87dbe4-de48-4ff6-bb84-cac7b1719661\") " pod="openshift-monitoring/node-exporter-9q5tr"
Jul 30 17:01:31.792221 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:31.792163    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ed87dbe4-de48-4ff6-bb84-cac7b1719661-metrics-client-ca\") pod \"node-exporter-9q5tr\" (UID: \"ed87dbe4-de48-4ff6-bb84-cac7b1719661\") " pod="openshift-monitoring/node-exporter-9q5tr"
Jul 30 17:01:31.792523 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:31.792317    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ed87dbe4-de48-4ff6-bb84-cac7b1719661-node-exporter-textfile\") pod \"node-exporter-9q5tr\" (UID: \"ed87dbe4-de48-4ff6-bb84-cac7b1719661\") " pod="openshift-monitoring/node-exporter-9q5tr"
Jul 30 17:01:31.792523 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:31.792376    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnpmz\" (UniqueName: \"kubernetes.io/projected/ed87dbe4-de48-4ff6-bb84-cac7b1719661-kube-api-access-nnpmz\") pod \"node-exporter-9q5tr\" (UID: \"ed87dbe4-de48-4ff6-bb84-cac7b1719661\") " pod="openshift-monitoring/node-exporter-9q5tr"
Jul 30 17:01:31.871619 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:31.871591    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9hlj5"
Jul 30 17:01:31.880726 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:31.880703    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4wpnm\""
Jul 30 17:01:31.881488 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:31.881454    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:31.889263 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:31.889244    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-55hfh"
Jul 30 17:01:31.893923 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:31.893506    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ed87dbe4-de48-4ff6-bb84-cac7b1719661-root\") pod \"node-exporter-9q5tr\" (UID: \"ed87dbe4-de48-4ff6-bb84-cac7b1719661\") " pod="openshift-monitoring/node-exporter-9q5tr"
Jul 30 17:01:31.893923 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:31.893549    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ed87dbe4-de48-4ff6-bb84-cac7b1719661-node-exporter-accelerators-collector-config\") pod \"node-exporter-9q5tr\" (UID: \"ed87dbe4-de48-4ff6-bb84-cac7b1719661\") " pod="openshift-monitoring/node-exporter-9q5tr"
Jul 30 17:01:31.893923 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:31.893586    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/45cb4db9-0f93-4fcb-80c5-20ff7c527aab-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-qvrdk\" (UID: \"45cb4db9-0f93-4fcb-80c5-20ff7c527aab\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-qvrdk"
Jul 30 17:01:31.893923 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:31.893615    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ed87dbe4-de48-4ff6-bb84-cac7b1719661-node-exporter-wtmp\") pod \"node-exporter-9q5tr\" (UID: \"ed87dbe4-de48-4ff6-bb84-cac7b1719661\") " pod="openshift-monitoring/node-exporter-9q5tr"
Jul 30 17:01:31.893923 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:31.893649    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ed87dbe4-de48-4ff6-bb84-cac7b1719661-sys\") pod \"node-exporter-9q5tr\" (UID: \"ed87dbe4-de48-4ff6-bb84-cac7b1719661\") " pod="openshift-monitoring/node-exporter-9q5tr"
Jul 30 17:01:31.893923 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:31.893682    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ed87dbe4-de48-4ff6-bb84-cac7b1719661-metrics-client-ca\") pod \"node-exporter-9q5tr\" (UID: \"ed87dbe4-de48-4ff6-bb84-cac7b1719661\") " pod="openshift-monitoring/node-exporter-9q5tr"
Jul 30 17:01:31.894402 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:31.894257    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ed87dbe4-de48-4ff6-bb84-cac7b1719661-node-exporter-accelerators-collector-config\") pod \"node-exporter-9q5tr\" (UID: \"ed87dbe4-de48-4ff6-bb84-cac7b1719661\") " pod="openshift-monitoring/node-exporter-9q5tr"
Jul 30 17:01:31.894402 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:31.894335    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ed87dbe4-de48-4ff6-bb84-cac7b1719661-node-exporter-textfile\") pod \"node-exporter-9q5tr\" (UID: \"ed87dbe4-de48-4ff6-bb84-cac7b1719661\") " pod="openshift-monitoring/node-exporter-9q5tr"
Jul 30 17:01:31.894402 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:31.894368    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnpmz\" (UniqueName: \"kubernetes.io/projected/ed87dbe4-de48-4ff6-bb84-cac7b1719661-kube-api-access-nnpmz\") pod \"node-exporter-9q5tr\" (UID: \"ed87dbe4-de48-4ff6-bb84-cac7b1719661\") " pod="openshift-monitoring/node-exporter-9q5tr"
Jul 30 17:01:31.894402 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:31.894399    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45cb4db9-0f93-4fcb-80c5-20ff7c527aab-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-qvrdk\" (UID: \"45cb4db9-0f93-4fcb-80c5-20ff7c527aab\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-qvrdk"
Jul 30 17:01:31.894602 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:01:31.894407    2555 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Jul 30 17:01:31.894602 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:31.894414    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cpbg\" (UniqueName: \"kubernetes.io/projected/45cb4db9-0f93-4fcb-80c5-20ff7c527aab-kube-api-access-2cpbg\") pod \"openshift-state-metrics-b8f49f5c6-qvrdk\" (UID: \"45cb4db9-0f93-4fcb-80c5-20ff7c527aab\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-qvrdk"
Jul 30 17:01:31.894602 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:31.894431    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ed87dbe4-de48-4ff6-bb84-cac7b1719661-node-exporter-tls\") pod \"node-exporter-9q5tr\" (UID: \"ed87dbe4-de48-4ff6-bb84-cac7b1719661\") " pod="openshift-monitoring/node-exporter-9q5tr"
Jul 30 17:01:31.894602 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:31.894449    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/45cb4db9-0f93-4fcb-80c5-20ff7c527aab-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-qvrdk\" (UID: \"45cb4db9-0f93-4fcb-80c5-20ff7c527aab\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-qvrdk"
Jul 30 17:01:31.894602 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:01:31.894469    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45cb4db9-0f93-4fcb-80c5-20ff7c527aab-openshift-state-metrics-tls podName:45cb4db9-0f93-4fcb-80c5-20ff7c527aab nodeName:}" failed. No retries permitted until 2026-07-30 17:01:32.394447814 +0000 UTC m=+66.881815208 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/45cb4db9-0f93-4fcb-80c5-20ff7c527aab-openshift-state-metrics-tls") pod "openshift-state-metrics-b8f49f5c6-qvrdk" (UID: "45cb4db9-0f93-4fcb-80c5-20ff7c527aab") : secret "openshift-state-metrics-tls" not found
Jul 30 17:01:31.894602 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:31.894511    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ed87dbe4-de48-4ff6-bb84-cac7b1719661-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9q5tr\" (UID: \"ed87dbe4-de48-4ff6-bb84-cac7b1719661\") " pod="openshift-monitoring/node-exporter-9q5tr"
Jul 30 17:01:31.894893 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:31.894334    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ed87dbe4-de48-4ff6-bb84-cac7b1719661-root\") pod \"node-exporter-9q5tr\" (UID: \"ed87dbe4-de48-4ff6-bb84-cac7b1719661\") " pod="openshift-monitoring/node-exporter-9q5tr"
Jul 30 17:01:31.894893 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:31.894834    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ed87dbe4-de48-4ff6-bb84-cac7b1719661-node-exporter-wtmp\") pod \"node-exporter-9q5tr\" (UID: \"ed87dbe4-de48-4ff6-bb84-cac7b1719661\") " pod="openshift-monitoring/node-exporter-9q5tr"
Jul 30 17:01:31.894893 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:31.894864    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ed87dbe4-de48-4ff6-bb84-cac7b1719661-sys\") pod \"node-exporter-9q5tr\" (UID: \"ed87dbe4-de48-4ff6-bb84-cac7b1719661\") " pod="openshift-monitoring/node-exporter-9q5tr"
Jul 30 17:01:31.895877 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:31.895286    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ed87dbe4-de48-4ff6-bb84-cac7b1719661-metrics-client-ca\") pod \"node-exporter-9q5tr\" (UID: \"ed87dbe4-de48-4ff6-bb84-cac7b1719661\") " pod="openshift-monitoring/node-exporter-9q5tr"
Jul 30 17:01:31.895877 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:01:31.895614    2555 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 30 17:01:31.895877 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:01:31.895664    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed87dbe4-de48-4ff6-bb84-cac7b1719661-node-exporter-tls podName:ed87dbe4-de48-4ff6-bb84-cac7b1719661 nodeName:}" failed. No retries permitted until 2026-07-30 17:01:32.395647204 +0000 UTC m=+66.883014586 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/ed87dbe4-de48-4ff6-bb84-cac7b1719661-node-exporter-tls") pod "node-exporter-9q5tr" (UID: "ed87dbe4-de48-4ff6-bb84-cac7b1719661") : secret "node-exporter-tls" not found
Jul 30 17:01:31.895877 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:31.895764    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45cb4db9-0f93-4fcb-80c5-20ff7c527aab-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-qvrdk\" (UID: \"45cb4db9-0f93-4fcb-80c5-20ff7c527aab\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-qvrdk"
Jul 30 17:01:31.896167 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:31.895927    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ed87dbe4-de48-4ff6-bb84-cac7b1719661-node-exporter-textfile\") pod \"node-exporter-9q5tr\" (UID: \"ed87dbe4-de48-4ff6-bb84-cac7b1719661\") " pod="openshift-monitoring/node-exporter-9q5tr"
Jul 30 17:01:31.897264 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:31.897237    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/45cb4db9-0f93-4fcb-80c5-20ff7c527aab-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-qvrdk\" (UID: \"45cb4db9-0f93-4fcb-80c5-20ff7c527aab\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-qvrdk"
Jul 30 17:01:31.897404 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:31.897382    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ed87dbe4-de48-4ff6-bb84-cac7b1719661-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9q5tr\" (UID: \"ed87dbe4-de48-4ff6-bb84-cac7b1719661\") " pod="openshift-monitoring/node-exporter-9q5tr"
Jul 30 17:01:31.905298 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:31.905247    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cpbg\" (UniqueName: \"kubernetes.io/projected/45cb4db9-0f93-4fcb-80c5-20ff7c527aab-kube-api-access-2cpbg\") pod \"openshift-state-metrics-b8f49f5c6-qvrdk\" (UID: \"45cb4db9-0f93-4fcb-80c5-20ff7c527aab\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-qvrdk"
Jul 30 17:01:31.905298 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:31.905275    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnpmz\" (UniqueName: \"kubernetes.io/projected/ed87dbe4-de48-4ff6-bb84-cac7b1719661-kube-api-access-nnpmz\") pod \"node-exporter-9q5tr\" (UID: \"ed87dbe4-de48-4ff6-bb84-cac7b1719661\") " pod="openshift-monitoring/node-exporter-9q5tr"
Jul 30 17:01:31.924755 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:31.924453    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qvgrv"]
Jul 30 17:01:31.927598 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:01:31.927570    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc0a3112_260e_47c8_87fd_bf3666ea17d0.slice/crio-96562d3875273a481c595519adab668cf1cab67c3ec205b48dfe2aa06e695213 WatchSource:0}: Error finding container 96562d3875273a481c595519adab668cf1cab67c3ec205b48dfe2aa06e695213: Status 404 returned error can't find the container with id 96562d3875273a481c595519adab668cf1cab67c3ec205b48dfe2aa06e695213
Jul 30 17:01:31.980643 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:31.980549    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:32.033748 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:32.033668    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9hlj5"]
Jul 30 17:01:32.036937 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:01:32.036903    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51afa6bb_724f_47ab_b47b_36dc8a61ca25.slice/crio-186c7d2b82c4c41fa4778ae2bcde0db21cbfccd1dcafb5686db20617e23f4ca9 WatchSource:0}: Error finding container 186c7d2b82c4c41fa4778ae2bcde0db21cbfccd1dcafb5686db20617e23f4ca9: Status 404 returned error can't find the container with id 186c7d2b82c4c41fa4778ae2bcde0db21cbfccd1dcafb5686db20617e23f4ca9
Jul 30 17:01:32.059471 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:32.059440    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-55hfh"]
Jul 30 17:01:32.061580 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:32.061554    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:32.062117 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:01:32.062088    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode85f9d71_fe3d_49d8_93e2_e4ac101603c7.slice/crio-692bc3dc374be284d5be78f13b4d35345519b6fe72daafaf7cb25acb71a5edde WatchSource:0}: Error finding container 692bc3dc374be284d5be78f13b4d35345519b6fe72daafaf7cb25acb71a5edde: Status 404 returned error can't find the container with id 692bc3dc374be284d5be78f13b4d35345519b6fe72daafaf7cb25acb71a5edde
Jul 30 17:01:32.132801 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:32.132741    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:32.230675 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:32.230599    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:32.330896 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:32.330866    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:32.397772 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:32.397734    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9hlj5" event={"ID":"51afa6bb-724f-47ab-b47b-36dc8a61ca25","Type":"ContainerStarted","Data":"186c7d2b82c4c41fa4778ae2bcde0db21cbfccd1dcafb5686db20617e23f4ca9"}
Jul 30 17:01:32.398282 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:32.398260    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/45cb4db9-0f93-4fcb-80c5-20ff7c527aab-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-qvrdk\" (UID: \"45cb4db9-0f93-4fcb-80c5-20ff7c527aab\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-qvrdk"
Jul 30 17:01:32.398428 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:32.398333    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ed87dbe4-de48-4ff6-bb84-cac7b1719661-node-exporter-tls\") pod \"node-exporter-9q5tr\" (UID: \"ed87dbe4-de48-4ff6-bb84-cac7b1719661\") " pod="openshift-monitoring/node-exporter-9q5tr"
Jul 30 17:01:32.399426 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:32.399362    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-55hfh" event={"ID":"e85f9d71-fe3d-49d8-93e2-e4ac101603c7","Type":"ContainerStarted","Data":"692bc3dc374be284d5be78f13b4d35345519b6fe72daafaf7cb25acb71a5edde"}
Jul 30 17:01:32.400659 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:32.400634    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qvgrv" event={"ID":"fc0a3112-260e-47c8-87fd-bf3666ea17d0","Type":"ContainerStarted","Data":"96562d3875273a481c595519adab668cf1cab67c3ec205b48dfe2aa06e695213"}
Jul 30 17:01:32.400989 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:32.400968    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ed87dbe4-de48-4ff6-bb84-cac7b1719661-node-exporter-tls\") pod \"node-exporter-9q5tr\" (UID: \"ed87dbe4-de48-4ff6-bb84-cac7b1719661\") " pod="openshift-monitoring/node-exporter-9q5tr"
Jul 30 17:01:32.401092 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:32.401040    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/45cb4db9-0f93-4fcb-80c5-20ff7c527aab-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-qvrdk\" (UID: \"45cb4db9-0f93-4fcb-80c5-20ff7c527aab\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-qvrdk"
Jul 30 17:01:32.632907 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:32.632860    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-qvrdk"
Jul 30 17:01:32.656923 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:32.656691    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9q5tr"
Jul 30 17:01:32.712853 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:32.712757    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:32.803806 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:32.803551    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-qvrdk"]
Jul 30 17:01:32.803806 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:32.803750    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:32.810517 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:01:32.810484    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45cb4db9_0f93_4fcb_80c5_20ff7c527aab.slice/crio-5e6b41601791e55ea7f31cdbc8691e0248ee12cff6fb8d988fcebc0fc1e7bac5 WatchSource:0}: Error finding container 5e6b41601791e55ea7f31cdbc8691e0248ee12cff6fb8d988fcebc0fc1e7bac5: Status 404 returned error can't find the container with id 5e6b41601791e55ea7f31cdbc8691e0248ee12cff6fb8d988fcebc0fc1e7bac5
Jul 30 17:01:32.933783 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:32.933688    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:32.999346 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:32.999282    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:33.407927 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:33.407837    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9q5tr" event={"ID":"ed87dbe4-de48-4ff6-bb84-cac7b1719661","Type":"ContainerStarted","Data":"263d9cd989920fd90570505410029c18c98d410ae33bc92a3a162ed1d33a7f00"}
Jul 30 17:01:33.410627 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:33.410481    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-qvrdk" event={"ID":"45cb4db9-0f93-4fcb-80c5-20ff7c527aab","Type":"ContainerStarted","Data":"5aa9debad916aebd6e4ef0279a8a7e7421c70cdea522d8852725ea7ee194420c"}
Jul 30 17:01:33.410627 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:33.410547    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-qvrdk" event={"ID":"45cb4db9-0f93-4fcb-80c5-20ff7c527aab","Type":"ContainerStarted","Data":"17821bc0014a1274ddd8ede974a1b10d44984eb0b86c8db382e127f784acce5d"}
Jul 30 17:01:33.410627 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:33.410578    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-qvrdk" event={"ID":"45cb4db9-0f93-4fcb-80c5-20ff7c527aab","Type":"ContainerStarted","Data":"5e6b41601791e55ea7f31cdbc8691e0248ee12cff6fb8d988fcebc0fc1e7bac5"}
Jul 30 17:01:33.822811 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:33.822608    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6448464666-hls4h"]
Jul 30 17:01:33.826834 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:33.826801    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6448464666-hls4h"
Jul 30 17:01:33.831664 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:33.831383    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-2ns2i2umm9gb4\""
Jul 30 17:01:33.833383 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:33.832188    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 30 17:01:33.833383 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:33.832637    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-zptgv\""
Jul 30 17:01:33.833383 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:33.832866    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 30 17:01:33.833383 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:33.832937    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 30 17:01:33.833383 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:33.833083    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 30 17:01:33.833383 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:33.833229    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 30 17:01:33.858458 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:33.858422    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6448464666-hls4h"]
Jul 30 17:01:33.912953 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:33.912490    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/266b5284-fb72-46c4-a694-37535e997cd3-secret-grpc-tls\") pod \"thanos-querier-6448464666-hls4h\" (UID: \"266b5284-fb72-46c4-a694-37535e997cd3\") " pod="openshift-monitoring/thanos-querier-6448464666-hls4h"
Jul 30 17:01:33.912953 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:33.912538    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8xqb\" (UniqueName: \"kubernetes.io/projected/266b5284-fb72-46c4-a694-37535e997cd3-kube-api-access-q8xqb\") pod \"thanos-querier-6448464666-hls4h\" (UID: \"266b5284-fb72-46c4-a694-37535e997cd3\") " pod="openshift-monitoring/thanos-querier-6448464666-hls4h"
Jul 30 17:01:33.912953 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:33.912593    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/266b5284-fb72-46c4-a694-37535e997cd3-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6448464666-hls4h\" (UID: \"266b5284-fb72-46c4-a694-37535e997cd3\") " pod="openshift-monitoring/thanos-querier-6448464666-hls4h"
Jul 30 17:01:33.912953 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:33.912634    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/266b5284-fb72-46c4-a694-37535e997cd3-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6448464666-hls4h\" (UID: \"266b5284-fb72-46c4-a694-37535e997cd3\") " pod="openshift-monitoring/thanos-querier-6448464666-hls4h"
Jul 30 17:01:33.912953 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:33.912680    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/266b5284-fb72-46c4-a694-37535e997cd3-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6448464666-hls4h\" (UID: \"266b5284-fb72-46c4-a694-37535e997cd3\") " pod="openshift-monitoring/thanos-querier-6448464666-hls4h"
Jul 30 17:01:33.912953 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:33.912713    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/266b5284-fb72-46c4-a694-37535e997cd3-secret-thanos-querier-tls\") pod \"thanos-querier-6448464666-hls4h\" (UID: \"266b5284-fb72-46c4-a694-37535e997cd3\") " pod="openshift-monitoring/thanos-querier-6448464666-hls4h"
Jul 30 17:01:33.912953 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:33.912767    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/266b5284-fb72-46c4-a694-37535e997cd3-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6448464666-hls4h\" (UID: \"266b5284-fb72-46c4-a694-37535e997cd3\") " pod="openshift-monitoring/thanos-querier-6448464666-hls4h"
Jul 30 17:01:33.912953 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:33.912799    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/266b5284-fb72-46c4-a694-37535e997cd3-metrics-client-ca\") pod \"thanos-querier-6448464666-hls4h\" (UID: \"266b5284-fb72-46c4-a694-37535e997cd3\") " pod="openshift-monitoring/thanos-querier-6448464666-hls4h"
Jul 30 17:01:34.014321 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:34.014034    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/266b5284-fb72-46c4-a694-37535e997cd3-secret-grpc-tls\") pod \"thanos-querier-6448464666-hls4h\" (UID: \"266b5284-fb72-46c4-a694-37535e997cd3\") " pod="openshift-monitoring/thanos-querier-6448464666-hls4h"
Jul 30 17:01:34.014321 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:34.014080    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8xqb\" (UniqueName: \"kubernetes.io/projected/266b5284-fb72-46c4-a694-37535e997cd3-kube-api-access-q8xqb\") pod \"thanos-querier-6448464666-hls4h\" (UID: \"266b5284-fb72-46c4-a694-37535e997cd3\") " pod="openshift-monitoring/thanos-querier-6448464666-hls4h"
Jul 30 17:01:34.014321 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:34.014118    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/266b5284-fb72-46c4-a694-37535e997cd3-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6448464666-hls4h\" (UID: \"266b5284-fb72-46c4-a694-37535e997cd3\") " pod="openshift-monitoring/thanos-querier-6448464666-hls4h"
Jul 30 17:01:34.015668 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:34.014642    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/266b5284-fb72-46c4-a694-37535e997cd3-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6448464666-hls4h\" (UID: \"266b5284-fb72-46c4-a694-37535e997cd3\") " pod="openshift-monitoring/thanos-querier-6448464666-hls4h"
Jul 30 17:01:34.015668 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:34.014685    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/266b5284-fb72-46c4-a694-37535e997cd3-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6448464666-hls4h\" (UID: \"266b5284-fb72-46c4-a694-37535e997cd3\") " pod="openshift-monitoring/thanos-querier-6448464666-hls4h"
Jul 30 17:01:34.015668 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:34.014720    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/266b5284-fb72-46c4-a694-37535e997cd3-secret-thanos-querier-tls\") pod \"thanos-querier-6448464666-hls4h\" (UID: \"266b5284-fb72-46c4-a694-37535e997cd3\") " pod="openshift-monitoring/thanos-querier-6448464666-hls4h"
Jul 30 17:01:34.015668 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:34.014772    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/266b5284-fb72-46c4-a694-37535e997cd3-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6448464666-hls4h\" (UID: \"266b5284-fb72-46c4-a694-37535e997cd3\") " pod="openshift-monitoring/thanos-querier-6448464666-hls4h"
Jul 30 17:01:34.015668 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:34.014802    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/266b5284-fb72-46c4-a694-37535e997cd3-metrics-client-ca\") pod \"thanos-querier-6448464666-hls4h\" (UID: \"266b5284-fb72-46c4-a694-37535e997cd3\") " pod="openshift-monitoring/thanos-querier-6448464666-hls4h"
Jul 30 17:01:34.015668 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:34.015576    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/266b5284-fb72-46c4-a694-37535e997cd3-metrics-client-ca\") pod \"thanos-querier-6448464666-hls4h\" (UID: \"266b5284-fb72-46c4-a694-37535e997cd3\") " pod="openshift-monitoring/thanos-querier-6448464666-hls4h"
Jul 30 17:01:34.021904 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:34.021843    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/266b5284-fb72-46c4-a694-37535e997cd3-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6448464666-hls4h\" (UID: \"266b5284-fb72-46c4-a694-37535e997cd3\") " pod="openshift-monitoring/thanos-querier-6448464666-hls4h"
Jul 30 17:01:34.024353 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:34.024309    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8xqb\" (UniqueName: \"kubernetes.io/projected/266b5284-fb72-46c4-a694-37535e997cd3-kube-api-access-q8xqb\") pod \"thanos-querier-6448464666-hls4h\" (UID: \"266b5284-fb72-46c4-a694-37535e997cd3\") " pod="openshift-monitoring/thanos-querier-6448464666-hls4h"
Jul 30 17:01:34.025875 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:34.025816    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/266b5284-fb72-46c4-a694-37535e997cd3-secret-thanos-querier-tls\") pod \"thanos-querier-6448464666-hls4h\" (UID: \"266b5284-fb72-46c4-a694-37535e997cd3\") " pod="openshift-monitoring/thanos-querier-6448464666-hls4h"
Jul 30 17:01:34.026103 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:34.026076    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/266b5284-fb72-46c4-a694-37535e997cd3-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6448464666-hls4h\" (UID: \"266b5284-fb72-46c4-a694-37535e997cd3\") " pod="openshift-monitoring/thanos-querier-6448464666-hls4h"
Jul 30 17:01:34.026626 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:34.026603    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/266b5284-fb72-46c4-a694-37535e997cd3-secret-grpc-tls\") pod \"thanos-querier-6448464666-hls4h\" (UID: \"266b5284-fb72-46c4-a694-37535e997cd3\") " pod="openshift-monitoring/thanos-querier-6448464666-hls4h"
Jul 30 17:01:34.027460 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:34.027434    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/266b5284-fb72-46c4-a694-37535e997cd3-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6448464666-hls4h\" (UID: \"266b5284-fb72-46c4-a694-37535e997cd3\") " pod="openshift-monitoring/thanos-querier-6448464666-hls4h"
Jul 30 17:01:34.028084 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:34.027954    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/266b5284-fb72-46c4-a694-37535e997cd3-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6448464666-hls4h\" (UID: \"266b5284-fb72-46c4-a694-37535e997cd3\") " pod="openshift-monitoring/thanos-querier-6448464666-hls4h"
Jul 30 17:01:34.141177 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:34.141125    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6448464666-hls4h"
Jul 30 17:01:34.338121 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:34.338058    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7675687bb6-jkh86"]
Jul 30 17:01:34.342199 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:34.342152    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:34.367857 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:34.367826    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7675687bb6-jkh86"]
Jul 30 17:01:34.368093 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:34.367948    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7675687bb6-jkh86"
Jul 30 17:01:34.372032 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:34.371737    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 30 17:01:34.372032 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:34.371792    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-5rx9s\""
Jul 30 17:01:34.372032 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:34.371959    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 30 17:01:34.372838 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:34.372785    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 30 17:01:34.373346 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:34.373083    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 30 17:01:34.373346 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:34.373151    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 30 17:01:34.379137 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:34.379112    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 30 17:01:34.418500 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:34.418136    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f4fb739f-b8d6-494d-987d-cf9def14829e-oauth-serving-cert\") pod \"console-7675687bb6-jkh86\" (UID: \"f4fb739f-b8d6-494d-987d-cf9def14829e\") " pod="openshift-console/console-7675687bb6-jkh86"
Jul 30 17:01:34.418500 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:34.418177    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f4fb739f-b8d6-494d-987d-cf9def14829e-console-config\") pod \"console-7675687bb6-jkh86\" (UID: \"f4fb739f-b8d6-494d-987d-cf9def14829e\") " pod="openshift-console/console-7675687bb6-jkh86"
Jul 30 17:01:34.418500 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:34.418218    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4fb739f-b8d6-494d-987d-cf9def14829e-trusted-ca-bundle\") pod \"console-7675687bb6-jkh86\" (UID: \"f4fb739f-b8d6-494d-987d-cf9def14829e\") " pod="openshift-console/console-7675687bb6-jkh86"
Jul 30 17:01:34.418500 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:34.418253    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f4fb739f-b8d6-494d-987d-cf9def14829e-service-ca\") pod \"console-7675687bb6-jkh86\" (UID: \"f4fb739f-b8d6-494d-987d-cf9def14829e\") " pod="openshift-console/console-7675687bb6-jkh86"
Jul 30 17:01:34.418500 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:34.418281    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f4fb739f-b8d6-494d-987d-cf9def14829e-console-serving-cert\") pod \"console-7675687bb6-jkh86\" (UID: \"f4fb739f-b8d6-494d-987d-cf9def14829e\") " pod="openshift-console/console-7675687bb6-jkh86"
Jul 30 17:01:34.418500 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:34.418311    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f4fb739f-b8d6-494d-987d-cf9def14829e-console-oauth-config\") pod \"console-7675687bb6-jkh86\" (UID: \"f4fb739f-b8d6-494d-987d-cf9def14829e\") " pod="openshift-console/console-7675687bb6-jkh86"
Jul 30 17:01:34.418500 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:34.418346    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sjf6\" (UniqueName: \"kubernetes.io/projected/f4fb739f-b8d6-494d-987d-cf9def14829e-kube-api-access-8sjf6\") pod \"console-7675687bb6-jkh86\" (UID: \"f4fb739f-b8d6-494d-987d-cf9def14829e\") " pod="openshift-console/console-7675687bb6-jkh86"
Jul 30 17:01:34.421548 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:34.421493    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-55hfh" event={"ID":"e85f9d71-fe3d-49d8-93e2-e4ac101603c7","Type":"ContainerStarted","Data":"0110f756f69e8d4dce6ff318d482741900b790ef295aa2b6f583b183a9871a0c"}
Jul 30 17:01:34.421662 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:34.421558    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-55hfh" event={"ID":"e85f9d71-fe3d-49d8-93e2-e4ac101603c7","Type":"ContainerStarted","Data":"5b8a127c6ad4f64ca074657444d44b8b61af2d146a529b2675b79703ae60fcf3"}
Jul 30 17:01:34.439742 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:34.439039    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-55hfh" podStartSLOduration=66.927010438 podStartE2EDuration="1m8.438998532s" totalImagesPullingTime="1.511988094s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:00:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:01:34.438560761 +0000 UTC m=+68.925928163" watchObservedRunningTime="2026-07-30 17:01:34.438998532 +0000 UTC m=+68.926365935"
Jul 30 17:01:34.442830 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:34.442800    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:34.520195 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:34.519609    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f4fb739f-b8d6-494d-987d-cf9def14829e-console-oauth-config\") pod \"console-7675687bb6-jkh86\" (UID: \"f4fb739f-b8d6-494d-987d-cf9def14829e\") " pod="openshift-console/console-7675687bb6-jkh86"
Jul 30 17:01:34.520195 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:34.519661    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sjf6\" (UniqueName: \"kubernetes.io/projected/f4fb739f-b8d6-494d-987d-cf9def14829e-kube-api-access-8sjf6\") pod \"console-7675687bb6-jkh86\" (UID: \"f4fb739f-b8d6-494d-987d-cf9def14829e\") " pod="openshift-console/console-7675687bb6-jkh86"
Jul 30 17:01:34.520195 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:34.519748    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f4fb739f-b8d6-494d-987d-cf9def14829e-oauth-serving-cert\") pod \"console-7675687bb6-jkh86\" (UID: \"f4fb739f-b8d6-494d-987d-cf9def14829e\") " pod="openshift-console/console-7675687bb6-jkh86"
Jul 30 17:01:34.520195 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:34.519790    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f4fb739f-b8d6-494d-987d-cf9def14829e-console-config\") pod \"console-7675687bb6-jkh86\" (UID: \"f4fb739f-b8d6-494d-987d-cf9def14829e\") " pod="openshift-console/console-7675687bb6-jkh86"
Jul 30 17:01:34.520195 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:34.519826    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4fb739f-b8d6-494d-987d-cf9def14829e-trusted-ca-bundle\") pod \"console-7675687bb6-jkh86\" (UID: \"f4fb739f-b8d6-494d-987d-cf9def14829e\") " pod="openshift-console/console-7675687bb6-jkh86"
Jul 30 17:01:34.521600 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:34.520705    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f4fb739f-b8d6-494d-987d-cf9def14829e-oauth-serving-cert\") pod \"console-7675687bb6-jkh86\" (UID: \"f4fb739f-b8d6-494d-987d-cf9def14829e\") " pod="openshift-console/console-7675687bb6-jkh86"
Jul 30 17:01:34.521600 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:34.520855    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f4fb739f-b8d6-494d-987d-cf9def14829e-console-config\") pod \"console-7675687bb6-jkh86\" (UID: \"f4fb739f-b8d6-494d-987d-cf9def14829e\") " pod="openshift-console/console-7675687bb6-jkh86"
Jul 30 17:01:34.521600 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:34.521362    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f4fb739f-b8d6-494d-987d-cf9def14829e-service-ca\") pod \"console-7675687bb6-jkh86\" (UID: \"f4fb739f-b8d6-494d-987d-cf9def14829e\") " pod="openshift-console/console-7675687bb6-jkh86"
Jul 30 17:01:34.521600 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:34.521410    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f4fb739f-b8d6-494d-987d-cf9def14829e-console-serving-cert\") pod \"console-7675687bb6-jkh86\" (UID: \"f4fb739f-b8d6-494d-987d-cf9def14829e\") " pod="openshift-console/console-7675687bb6-jkh86"
Jul 30 17:01:34.521600 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:34.521516    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4fb739f-b8d6-494d-987d-cf9def14829e-trusted-ca-bundle\") pod \"console-7675687bb6-jkh86\" (UID: \"f4fb739f-b8d6-494d-987d-cf9def14829e\") " pod="openshift-console/console-7675687bb6-jkh86"
Jul 30 17:01:34.522076 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:34.522020    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f4fb739f-b8d6-494d-987d-cf9def14829e-service-ca\") pod \"console-7675687bb6-jkh86\" (UID: \"f4fb739f-b8d6-494d-987d-cf9def14829e\") " pod="openshift-console/console-7675687bb6-jkh86"
Jul 30 17:01:34.523033 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:34.522983    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f4fb739f-b8d6-494d-987d-cf9def14829e-console-oauth-config\") pod \"console-7675687bb6-jkh86\" (UID: \"f4fb739f-b8d6-494d-987d-cf9def14829e\") " pod="openshift-console/console-7675687bb6-jkh86"
Jul 30 17:01:34.524846 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:34.524775    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f4fb739f-b8d6-494d-987d-cf9def14829e-console-serving-cert\") pod \"console-7675687bb6-jkh86\" (UID: \"f4fb739f-b8d6-494d-987d-cf9def14829e\") " pod="openshift-console/console-7675687bb6-jkh86"
Jul 30 17:01:34.531377 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:34.531345    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sjf6\" (UniqueName: \"kubernetes.io/projected/f4fb739f-b8d6-494d-987d-cf9def14829e-kube-api-access-8sjf6\") pod \"console-7675687bb6-jkh86\" (UID: \"f4fb739f-b8d6-494d-987d-cf9def14829e\") " pod="openshift-console/console-7675687bb6-jkh86"
Jul 30 17:01:34.685148 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:34.684882    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7675687bb6-jkh86"
Jul 30 17:01:34.779315 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:34.779228    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:34.784875 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:34.784774    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6448464666-hls4h"]
Jul 30 17:01:34.876084 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:34.875959    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:34.956363 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:34.955896    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:35.020679 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:35.020648    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:35.121726 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:35.121695    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:35.988746 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:35.988702    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:36.060831 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:36.060798    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7545856648-6x6pd"]
Jul 30 17:01:36.072925 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:36.072892    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7545856648-6x6pd"
Jul 30 17:01:36.073129 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:36.073107    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7545856648-6x6pd"]
Jul 30 17:01:36.077415 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:36.077394    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-5fl07f1r9emdq\""
Jul 30 17:01:36.077605 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:36.077583    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 30 17:01:36.077701 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:36.077463    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 30 17:01:36.077701 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:36.077401    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 30 17:01:36.077701 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:36.077631    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 30 17:01:36.077851 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:36.077502    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-hrxbf\""
Jul 30 17:01:36.133943 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:36.133909    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/42318d15-ab44-4938-930f-ee2961a6d00b-metrics-server-audit-profiles\") pod \"metrics-server-7545856648-6x6pd\" (UID: \"42318d15-ab44-4938-930f-ee2961a6d00b\") " pod="openshift-monitoring/metrics-server-7545856648-6x6pd"
Jul 30 17:01:36.133943 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:36.133949    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4zg8\" (UniqueName: \"kubernetes.io/projected/42318d15-ab44-4938-930f-ee2961a6d00b-kube-api-access-d4zg8\") pod \"metrics-server-7545856648-6x6pd\" (UID: \"42318d15-ab44-4938-930f-ee2961a6d00b\") " pod="openshift-monitoring/metrics-server-7545856648-6x6pd"
Jul 30 17:01:36.134177 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:36.133984    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/42318d15-ab44-4938-930f-ee2961a6d00b-secret-metrics-server-tls\") pod \"metrics-server-7545856648-6x6pd\" (UID: \"42318d15-ab44-4938-930f-ee2961a6d00b\") " pod="openshift-monitoring/metrics-server-7545856648-6x6pd"
Jul 30 17:01:36.134177 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:36.134060    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/42318d15-ab44-4938-930f-ee2961a6d00b-audit-log\") pod \"metrics-server-7545856648-6x6pd\" (UID: \"42318d15-ab44-4938-930f-ee2961a6d00b\") " pod="openshift-monitoring/metrics-server-7545856648-6x6pd"
Jul 30 17:01:36.134177 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:36.134094    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42318d15-ab44-4938-930f-ee2961a6d00b-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7545856648-6x6pd\" (UID: \"42318d15-ab44-4938-930f-ee2961a6d00b\") " pod="openshift-monitoring/metrics-server-7545856648-6x6pd"
Jul 30 17:01:36.134177 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:36.134126    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/42318d15-ab44-4938-930f-ee2961a6d00b-secret-metrics-server-client-certs\") pod \"metrics-server-7545856648-6x6pd\" (UID: \"42318d15-ab44-4938-930f-ee2961a6d00b\") " pod="openshift-monitoring/metrics-server-7545856648-6x6pd"
Jul 30 17:01:36.134177 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:36.134151    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/42318d15-ab44-4938-930f-ee2961a6d00b-client-ca-bundle\") pod \"metrics-server-7545856648-6x6pd\" (UID: \"42318d15-ab44-4938-930f-ee2961a6d00b\") " pod="openshift-monitoring/metrics-server-7545856648-6x6pd"
Jul 30 17:01:36.235030 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:36.234961    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/42318d15-ab44-4938-930f-ee2961a6d00b-client-ca-bundle\") pod \"metrics-server-7545856648-6x6pd\" (UID: \"42318d15-ab44-4938-930f-ee2961a6d00b\") " pod="openshift-monitoring/metrics-server-7545856648-6x6pd"
Jul 30 17:01:36.235218 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:36.235064    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/42318d15-ab44-4938-930f-ee2961a6d00b-metrics-server-audit-profiles\") pod \"metrics-server-7545856648-6x6pd\" (UID: \"42318d15-ab44-4938-930f-ee2961a6d00b\") " pod="openshift-monitoring/metrics-server-7545856648-6x6pd"
Jul 30 17:01:36.235285 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:36.235201    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4zg8\" (UniqueName: \"kubernetes.io/projected/42318d15-ab44-4938-930f-ee2961a6d00b-kube-api-access-d4zg8\") pod \"metrics-server-7545856648-6x6pd\" (UID: \"42318d15-ab44-4938-930f-ee2961a6d00b\") " pod="openshift-monitoring/metrics-server-7545856648-6x6pd"
Jul 30 17:01:36.235341 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:36.235280    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/42318d15-ab44-4938-930f-ee2961a6d00b-secret-metrics-server-tls\") pod \"metrics-server-7545856648-6x6pd\" (UID: \"42318d15-ab44-4938-930f-ee2961a6d00b\") " pod="openshift-monitoring/metrics-server-7545856648-6x6pd"
Jul 30 17:01:36.235341 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:36.235331    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/42318d15-ab44-4938-930f-ee2961a6d00b-audit-log\") pod \"metrics-server-7545856648-6x6pd\" (UID: \"42318d15-ab44-4938-930f-ee2961a6d00b\") " pod="openshift-monitoring/metrics-server-7545856648-6x6pd"
Jul 30 17:01:36.235470 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:36.235369    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42318d15-ab44-4938-930f-ee2961a6d00b-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7545856648-6x6pd\" (UID: \"42318d15-ab44-4938-930f-ee2961a6d00b\") " pod="openshift-monitoring/metrics-server-7545856648-6x6pd"
Jul 30 17:01:36.235470 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:36.235401    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/42318d15-ab44-4938-930f-ee2961a6d00b-secret-metrics-server-client-certs\") pod \"metrics-server-7545856648-6x6pd\" (UID: \"42318d15-ab44-4938-930f-ee2961a6d00b\") " pod="openshift-monitoring/metrics-server-7545856648-6x6pd"
Jul 30 17:01:36.236027 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:36.235746    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/42318d15-ab44-4938-930f-ee2961a6d00b-audit-log\") pod \"metrics-server-7545856648-6x6pd\" (UID: \"42318d15-ab44-4938-930f-ee2961a6d00b\") " pod="openshift-monitoring/metrics-server-7545856648-6x6pd"
Jul 30 17:01:36.236210 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:36.236183    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/42318d15-ab44-4938-930f-ee2961a6d00b-metrics-server-audit-profiles\") pod \"metrics-server-7545856648-6x6pd\" (UID: \"42318d15-ab44-4938-930f-ee2961a6d00b\") " pod="openshift-monitoring/metrics-server-7545856648-6x6pd"
Jul 30 17:01:36.236301 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:36.236260    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42318d15-ab44-4938-930f-ee2961a6d00b-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7545856648-6x6pd\" (UID: \"42318d15-ab44-4938-930f-ee2961a6d00b\") " pod="openshift-monitoring/metrics-server-7545856648-6x6pd"
Jul 30 17:01:36.238281 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:36.238247    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/42318d15-ab44-4938-930f-ee2961a6d00b-client-ca-bundle\") pod \"metrics-server-7545856648-6x6pd\" (UID: \"42318d15-ab44-4938-930f-ee2961a6d00b\") " pod="openshift-monitoring/metrics-server-7545856648-6x6pd"
Jul 30 17:01:36.238398 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:36.238360    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/42318d15-ab44-4938-930f-ee2961a6d00b-secret-metrics-server-tls\") pod \"metrics-server-7545856648-6x6pd\" (UID: \"42318d15-ab44-4938-930f-ee2961a6d00b\") " pod="openshift-monitoring/metrics-server-7545856648-6x6pd"
Jul 30 17:01:36.238697 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:36.238676    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/42318d15-ab44-4938-930f-ee2961a6d00b-secret-metrics-server-client-certs\") pod \"metrics-server-7545856648-6x6pd\" (UID: \"42318d15-ab44-4938-930f-ee2961a6d00b\") " pod="openshift-monitoring/metrics-server-7545856648-6x6pd"
Jul 30 17:01:36.244301 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:36.244251    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4zg8\" (UniqueName: \"kubernetes.io/projected/42318d15-ab44-4938-930f-ee2961a6d00b-kube-api-access-d4zg8\") pod \"metrics-server-7545856648-6x6pd\" (UID: \"42318d15-ab44-4938-930f-ee2961a6d00b\") " pod="openshift-monitoring/metrics-server-7545856648-6x6pd"
Jul 30 17:01:36.379301 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:01:36.379261    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod266b5284_fb72_46c4_a694_37535e997cd3.slice/crio-97d552bae999edd25cfa53962ec5786d35788c07ccf7cf34af25a9da711040b5 WatchSource:0}: Error finding container 97d552bae999edd25cfa53962ec5786d35788c07ccf7cf34af25a9da711040b5: Status 404 returned error can't find the container with id 97d552bae999edd25cfa53962ec5786d35788c07ccf7cf34af25a9da711040b5
Jul 30 17:01:36.386392 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:36.386366    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7545856648-6x6pd"
Jul 30 17:01:36.432319 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:36.432286    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6448464666-hls4h" event={"ID":"266b5284-fb72-46c4-a694-37535e997cd3","Type":"ContainerStarted","Data":"97d552bae999edd25cfa53962ec5786d35788c07ccf7cf34af25a9da711040b5"}
Jul 30 17:01:36.498306 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:36.498217    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:36.508758 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:36.508730    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-dlr24"]
Jul 30 17:01:36.511459 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:36.511436    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-dlr24"
Jul 30 17:01:36.514049 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:36.513895    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 30 17:01:36.514049 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:36.514026    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-7g6d4\""
Jul 30 17:01:36.520617 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:36.520310    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-dlr24"]
Jul 30 17:01:36.596949 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:36.596907    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:36.638997 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:36.638964    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/411dc2d0-e97f-4574-abe7-1e06f22653f9-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-dlr24\" (UID: \"411dc2d0-e97f-4574-abe7-1e06f22653f9\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-dlr24"
Jul 30 17:01:36.639165 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:36.639030    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj68h\" (UniqueName: \"kubernetes.io/projected/411dc2d0-e97f-4574-abe7-1e06f22653f9-kube-api-access-tj68h\") pod \"monitoring-plugin-68b7fc9574-dlr24\" (UID: \"411dc2d0-e97f-4574-abe7-1e06f22653f9\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-dlr24"
Jul 30 17:01:36.660517 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:36.660480    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:36.739696 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:36.739662    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/411dc2d0-e97f-4574-abe7-1e06f22653f9-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-dlr24\" (UID: \"411dc2d0-e97f-4574-abe7-1e06f22653f9\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-dlr24"
Jul 30 17:01:36.739864 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:36.739728    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj68h\" (UniqueName: \"kubernetes.io/projected/411dc2d0-e97f-4574-abe7-1e06f22653f9-kube-api-access-tj68h\") pod \"monitoring-plugin-68b7fc9574-dlr24\" (UID: \"411dc2d0-e97f-4574-abe7-1e06f22653f9\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-dlr24"
Jul 30 17:01:36.742681 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:36.742647    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/411dc2d0-e97f-4574-abe7-1e06f22653f9-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-dlr24\" (UID: \"411dc2d0-e97f-4574-abe7-1e06f22653f9\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-dlr24"
Jul 30 17:01:36.755405 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:36.755340    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj68h\" (UniqueName: \"kubernetes.io/projected/411dc2d0-e97f-4574-abe7-1e06f22653f9-kube-api-access-tj68h\") pod \"monitoring-plugin-68b7fc9574-dlr24\" (UID: \"411dc2d0-e97f-4574-abe7-1e06f22653f9\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-dlr24"
Jul 30 17:01:36.824779 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:36.824742    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-dlr24"
Jul 30 17:01:37.966820 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:37.966771    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:38.033588 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:38.033559    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:38.112680 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:38.112647    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7675687bb6-jkh86"]
Jul 30 17:01:39.283578 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:39.283537    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:39.605039 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:39.604997    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:39.882887 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:39.882772    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:39.934113 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:39.934084    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:40.166695 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:40.166612    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:40.505345 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:40.505198    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:41.104920 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:41.104868    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:41.159531 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:41.159500    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:42.379138 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:42.379100    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:42.540173 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:42.540131    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:42.641229 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:42.641137    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:43.741669 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:43.741640    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7675687bb6-jkh86"]
Jul 30 17:01:43.778713 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:43.778681    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-8447bbff85-pspxs"]
Jul 30 17:01:43.940616 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:43.940558    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8447bbff85-pspxs"]
Jul 30 17:01:43.940797 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:43.940687    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8447bbff85-pspxs"
Jul 30 17:01:44.011521 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:44.011438    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/137b1ef1-19fb-49d3-878b-b6b85f333998-trusted-ca-bundle\") pod \"console-8447bbff85-pspxs\" (UID: \"137b1ef1-19fb-49d3-878b-b6b85f333998\") " pod="openshift-console/console-8447bbff85-pspxs"
Jul 30 17:01:44.011521 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:44.011496    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/137b1ef1-19fb-49d3-878b-b6b85f333998-console-config\") pod \"console-8447bbff85-pspxs\" (UID: \"137b1ef1-19fb-49d3-878b-b6b85f333998\") " pod="openshift-console/console-8447bbff85-pspxs"
Jul 30 17:01:44.011729 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:44.011594    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/137b1ef1-19fb-49d3-878b-b6b85f333998-service-ca\") pod \"console-8447bbff85-pspxs\" (UID: \"137b1ef1-19fb-49d3-878b-b6b85f333998\") " pod="openshift-console/console-8447bbff85-pspxs"
Jul 30 17:01:44.011729 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:44.011700    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/137b1ef1-19fb-49d3-878b-b6b85f333998-console-oauth-config\") pod \"console-8447bbff85-pspxs\" (UID: \"137b1ef1-19fb-49d3-878b-b6b85f333998\") " pod="openshift-console/console-8447bbff85-pspxs"
Jul 30 17:01:44.011819 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:44.011733    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/137b1ef1-19fb-49d3-878b-b6b85f333998-oauth-serving-cert\") pod \"console-8447bbff85-pspxs\" (UID: \"137b1ef1-19fb-49d3-878b-b6b85f333998\") " pod="openshift-console/console-8447bbff85-pspxs"
Jul 30 17:01:44.011819 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:44.011762    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8n9w\" (UniqueName: \"kubernetes.io/projected/137b1ef1-19fb-49d3-878b-b6b85f333998-kube-api-access-g8n9w\") pod \"console-8447bbff85-pspxs\" (UID: \"137b1ef1-19fb-49d3-878b-b6b85f333998\") " pod="openshift-console/console-8447bbff85-pspxs"
Jul 30 17:01:44.011819 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:44.011800    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/137b1ef1-19fb-49d3-878b-b6b85f333998-console-serving-cert\") pod \"console-8447bbff85-pspxs\" (UID: \"137b1ef1-19fb-49d3-878b-b6b85f333998\") " pod="openshift-console/console-8447bbff85-pspxs"
Jul 30 17:01:44.112311 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:44.112281    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/137b1ef1-19fb-49d3-878b-b6b85f333998-trusted-ca-bundle\") pod \"console-8447bbff85-pspxs\" (UID: \"137b1ef1-19fb-49d3-878b-b6b85f333998\") " pod="openshift-console/console-8447bbff85-pspxs"
Jul 30 17:01:44.112509 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:44.112320    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/137b1ef1-19fb-49d3-878b-b6b85f333998-console-config\") pod \"console-8447bbff85-pspxs\" (UID: \"137b1ef1-19fb-49d3-878b-b6b85f333998\") " pod="openshift-console/console-8447bbff85-pspxs"
Jul 30 17:01:44.112509 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:44.112347    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/137b1ef1-19fb-49d3-878b-b6b85f333998-service-ca\") pod \"console-8447bbff85-pspxs\" (UID: \"137b1ef1-19fb-49d3-878b-b6b85f333998\") " pod="openshift-console/console-8447bbff85-pspxs"
Jul 30 17:01:44.112509 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:44.112398    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/137b1ef1-19fb-49d3-878b-b6b85f333998-console-oauth-config\") pod \"console-8447bbff85-pspxs\" (UID: \"137b1ef1-19fb-49d3-878b-b6b85f333998\") " pod="openshift-console/console-8447bbff85-pspxs"
Jul 30 17:01:44.112509 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:44.112428    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/137b1ef1-19fb-49d3-878b-b6b85f333998-oauth-serving-cert\") pod \"console-8447bbff85-pspxs\" (UID: \"137b1ef1-19fb-49d3-878b-b6b85f333998\") " pod="openshift-console/console-8447bbff85-pspxs"
Jul 30 17:01:44.112509 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:44.112459    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8n9w\" (UniqueName: \"kubernetes.io/projected/137b1ef1-19fb-49d3-878b-b6b85f333998-kube-api-access-g8n9w\") pod \"console-8447bbff85-pspxs\" (UID: \"137b1ef1-19fb-49d3-878b-b6b85f333998\") " pod="openshift-console/console-8447bbff85-pspxs"
Jul 30 17:01:44.112509 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:44.112483    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/137b1ef1-19fb-49d3-878b-b6b85f333998-console-serving-cert\") pod \"console-8447bbff85-pspxs\" (UID: \"137b1ef1-19fb-49d3-878b-b6b85f333998\") " pod="openshift-console/console-8447bbff85-pspxs"
Jul 30 17:01:44.113238 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:44.113188    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/137b1ef1-19fb-49d3-878b-b6b85f333998-console-config\") pod \"console-8447bbff85-pspxs\" (UID: \"137b1ef1-19fb-49d3-878b-b6b85f333998\") " pod="openshift-console/console-8447bbff85-pspxs"
Jul 30 17:01:44.113470 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:44.113447    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/137b1ef1-19fb-49d3-878b-b6b85f333998-oauth-serving-cert\") pod \"console-8447bbff85-pspxs\" (UID: \"137b1ef1-19fb-49d3-878b-b6b85f333998\") " pod="openshift-console/console-8447bbff85-pspxs"
Jul 30 17:01:44.113555 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:44.113484    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/137b1ef1-19fb-49d3-878b-b6b85f333998-service-ca\") pod \"console-8447bbff85-pspxs\" (UID: \"137b1ef1-19fb-49d3-878b-b6b85f333998\") " pod="openshift-console/console-8447bbff85-pspxs"
Jul 30 17:01:44.113597 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:44.113545    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/137b1ef1-19fb-49d3-878b-b6b85f333998-trusted-ca-bundle\") pod \"console-8447bbff85-pspxs\" (UID: \"137b1ef1-19fb-49d3-878b-b6b85f333998\") " pod="openshift-console/console-8447bbff85-pspxs"
Jul 30 17:01:44.115189 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:44.115166    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/137b1ef1-19fb-49d3-878b-b6b85f333998-console-serving-cert\") pod \"console-8447bbff85-pspxs\" (UID: \"137b1ef1-19fb-49d3-878b-b6b85f333998\") " pod="openshift-console/console-8447bbff85-pspxs"
Jul 30 17:01:44.115283 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:44.115266    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/137b1ef1-19fb-49d3-878b-b6b85f333998-console-oauth-config\") pod \"console-8447bbff85-pspxs\" (UID: \"137b1ef1-19fb-49d3-878b-b6b85f333998\") " pod="openshift-console/console-8447bbff85-pspxs"
Jul 30 17:01:44.121980 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:44.121951    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8n9w\" (UniqueName: \"kubernetes.io/projected/137b1ef1-19fb-49d3-878b-b6b85f333998-kube-api-access-g8n9w\") pod \"console-8447bbff85-pspxs\" (UID: \"137b1ef1-19fb-49d3-878b-b6b85f333998\") " pod="openshift-console/console-8447bbff85-pspxs"
Jul 30 17:01:44.251798 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:44.251759    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8447bbff85-pspxs"
Jul 30 17:01:45.021532 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:45.021498    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:45.120081 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:45.120048    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:45.574947 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:01:45.574914    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4fb739f_b8d6_494d_987d_cf9def14829e.slice/crio-e75554b666bc061b3ce5d8d4a146a7550fbe5c1dba6d229111d3654491fb5fe1 WatchSource:0}: Error finding container e75554b666bc061b3ce5d8d4a146a7550fbe5c1dba6d229111d3654491fb5fe1: Status 404 returned error can't find the container with id e75554b666bc061b3ce5d8d4a146a7550fbe5c1dba6d229111d3654491fb5fe1
Jul 30 17:01:45.834578 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:45.834541    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-dlr24"]
Jul 30 17:01:45.856100 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:45.855935    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7545856648-6x6pd"]
Jul 30 17:01:46.333044 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:01:46.332981    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42318d15_ab44_4938_930f_ee2961a6d00b.slice/crio-df5fd0aa0f6465fa725ee04de90e28789a4b3d4016e86966bd7a55ef87499a53 WatchSource:0}: Error finding container df5fd0aa0f6465fa725ee04de90e28789a4b3d4016e86966bd7a55ef87499a53: Status 404 returned error can't find the container with id df5fd0aa0f6465fa725ee04de90e28789a4b3d4016e86966bd7a55ef87499a53
Jul 30 17:01:46.334555 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:01:46.334478    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod411dc2d0_e97f_4574_abe7_1e06f22653f9.slice/crio-c43a435d98d3ab581cb2f12328607f4435727d59bc825c9c39cc67b3ea158b87 WatchSource:0}: Error finding container c43a435d98d3ab581cb2f12328607f4435727d59bc825c9c39cc67b3ea158b87: Status 404 returned error can't find the container with id c43a435d98d3ab581cb2f12328607f4435727d59bc825c9c39cc67b3ea158b87
Jul 30 17:01:46.472270 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:46.472226    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-dlr24" event={"ID":"411dc2d0-e97f-4574-abe7-1e06f22653f9","Type":"ContainerStarted","Data":"c43a435d98d3ab581cb2f12328607f4435727d59bc825c9c39cc67b3ea158b87"}
Jul 30 17:01:46.473538 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:46.473503    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7675687bb6-jkh86" event={"ID":"f4fb739f-b8d6-494d-987d-cf9def14829e","Type":"ContainerStarted","Data":"e75554b666bc061b3ce5d8d4a146a7550fbe5c1dba6d229111d3654491fb5fe1"}
Jul 30 17:01:46.474699 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:46.474669    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7545856648-6x6pd" event={"ID":"42318d15-ab44-4938-930f-ee2961a6d00b","Type":"ContainerStarted","Data":"df5fd0aa0f6465fa725ee04de90e28789a4b3d4016e86966bd7a55ef87499a53"}
Jul 30 17:01:46.477081 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:46.477054    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qvgrv" event={"ID":"fc0a3112-260e-47c8-87fd-bf3666ea17d0","Type":"ContainerStarted","Data":"54ca61faa132dd1a3e3e3bdb7360a5b3690f48a9a9eedcf8dd77a934265aaed1"}
Jul 30 17:01:46.479145 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:46.479117    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9q5tr" event={"ID":"ed87dbe4-de48-4ff6-bb84-cac7b1719661","Type":"ContainerStarted","Data":"2d88280f306e2c0d9502898974956ce123e5c7e0bfb09d8bcb04fda6e4bf3791"}
Jul 30 17:01:46.481399 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:46.481367    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-qvrdk" event={"ID":"45cb4db9-0f93-4fcb-80c5-20ff7c527aab","Type":"ContainerStarted","Data":"12337f76fb0754c3d11c1ea547b3194e5dbb78cce1eeb52089a0138f92a60e14"}
Jul 30 17:01:46.490641 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:46.490210    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8447bbff85-pspxs"]
Jul 30 17:01:46.492532 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:01:46.492509    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod137b1ef1_19fb_49d3_878b_b6b85f333998.slice/crio-40b463c54a68b545371f59ceb5526057adf08fc67b251db9f17d25ea6c52298d WatchSource:0}: Error finding container 40b463c54a68b545371f59ceb5526057adf08fc67b251db9f17d25ea6c52298d: Status 404 returned error can't find the container with id 40b463c54a68b545371f59ceb5526057adf08fc67b251db9f17d25ea6c52298d
Jul 30 17:01:46.518995 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:46.518942    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-qvrdk" podStartSLOduration=3.041480119 podStartE2EDuration="15.518922226s" totalImagesPullingTime="12.477442107s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:01:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:01:46.518912765 +0000 UTC m=+81.006280160" watchObservedRunningTime="2026-07-30 17:01:46.518922226 +0000 UTC m=+81.006289628"
Jul 30 17:01:47.491081 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:47.490926    2555 generic.go:356] "Generic (PLEG): container finished" podID="ed87dbe4-de48-4ff6-bb84-cac7b1719661" containerID="2d88280f306e2c0d9502898974956ce123e5c7e0bfb09d8bcb04fda6e4bf3791" exitCode=0
Jul 30 17:01:47.491714 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:47.491618    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9q5tr" event={"ID":"ed87dbe4-de48-4ff6-bb84-cac7b1719661","Type":"ContainerDied","Data":"2d88280f306e2c0d9502898974956ce123e5c7e0bfb09d8bcb04fda6e4bf3791"}
Jul 30 17:01:47.494444 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:47.494316    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-lgsc2" event={"ID":"7bb3cf69-d970-4823-b2f4-f9f726efaf9d","Type":"ContainerStarted","Data":"1b07bcfb8c5cc9944ad8c11533cd507898e85ffec2bcb5a82ecc7d30c59eed4f"}
Jul 30 17:01:47.494444 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:47.494382    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-lgsc2"
Jul 30 17:01:47.497816 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:47.497754    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9hlj5" event={"ID":"51afa6bb-724f-47ab-b47b-36dc8a61ca25","Type":"ContainerStarted","Data":"2664f6a1810fb9116723cd1ff7daaede24825a67a8ac8103f78121869529b255"}
Jul 30 17:01:47.502445 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:47.502419    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6448464666-hls4h" event={"ID":"266b5284-fb72-46c4-a694-37535e997cd3","Type":"ContainerStarted","Data":"d669503688be3857e38c1c4e8f6ce61799ba4ed6057c7ae92badc27f86c882df"}
Jul 30 17:01:47.502546 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:47.502492    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6448464666-hls4h" event={"ID":"266b5284-fb72-46c4-a694-37535e997cd3","Type":"ContainerStarted","Data":"09bcdd5ebb041929823feb2180723b505c8ce333cf14f33da144cbf2ca8daf87"}
Jul 30 17:01:47.502546 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:47.502533    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6448464666-hls4h" event={"ID":"266b5284-fb72-46c4-a694-37535e997cd3","Type":"ContainerStarted","Data":"812d523af2734443e3b52787cf5c5892f66e9285f10722e6fcf2d6b20841ee30"}
Jul 30 17:01:47.504646 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:47.504622    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8447bbff85-pspxs" event={"ID":"137b1ef1-19fb-49d3-878b-b6b85f333998","Type":"ContainerStarted","Data":"40b463c54a68b545371f59ceb5526057adf08fc67b251db9f17d25ea6c52298d"}
Jul 30 17:01:47.504995 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:47.504942    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qvgrv"
Jul 30 17:01:47.506677 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:47.506657    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-lgsc2"
Jul 30 17:01:47.535700 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:47.535615    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-lgsc2" podStartSLOduration=3.496155414 podStartE2EDuration="25.535593143s" totalImagesPullingTime="22.039437729s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:01:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:01:47.534519616 +0000 UTC m=+82.021887019" watchObservedRunningTime="2026-07-30 17:01:47.535593143 +0000 UTC m=+82.022960545"
Jul 30 17:01:47.571566 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:47.571509    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-9hlj5" podStartSLOduration=67.27741666 podStartE2EDuration="1m21.571487416s" totalImagesPullingTime="14.294070756s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 17:00:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:01:47.553515853 +0000 UTC m=+82.040883255" watchObservedRunningTime="2026-07-30 17:01:47.571487416 +0000 UTC m=+82.058854799"
Jul 30 17:01:48.365058 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:48.364981    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:48.464795 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:48.464682    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:48.516432 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:48.516403    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9q5tr" event={"ID":"ed87dbe4-de48-4ff6-bb84-cac7b1719661","Type":"ContainerStarted","Data":"31b1e8ed741d3ff394e1ed5b02ee12b5c6aec31a18392cc0ddc25d334284313b"}
Jul 30 17:01:48.517501 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:48.517460    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9q5tr" event={"ID":"ed87dbe4-de48-4ff6-bb84-cac7b1719661","Type":"ContainerStarted","Data":"ac1308cb1d0929b5442bad51f419b345fcd017b0b86467956245be49fb0b1a08"}
Jul 30 17:01:48.538127 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:48.538041    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-qvgrv" podStartSLOduration=68.8040393 podStartE2EDuration="1m22.538021387s" totalImagesPullingTime="13.733982087s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:00:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:01:47.572386974 +0000 UTC m=+82.059754376" watchObservedRunningTime="2026-07-30 17:01:48.538021387 +0000 UTC m=+83.025388780"
Jul 30 17:01:48.539103 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:48.539058    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-9q5tr" podStartSLOduration=4.671842172 podStartE2EDuration="17.539042564s" totalImagesPullingTime="12.774482291s" totalInitContainerRuntime="92.718101ms" isStatefulPod=true podCreationTimestamp="2026-07-30 17:01:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:01:48.537745519 +0000 UTC m=+83.025112922" watchObservedRunningTime="2026-07-30 17:01:48.539042564 +0000 UTC m=+83.026409966"
Jul 30 17:01:52.537836 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:52.537027    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7545856648-6x6pd" event={"ID":"42318d15-ab44-4938-930f-ee2961a6d00b","Type":"ContainerStarted","Data":"9bf07ceba2626f34eac4f685aaae1b2370d11851eceee67289cd19e34a267241"}
Jul 30 17:01:52.539064 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:52.539030    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8447bbff85-pspxs" event={"ID":"137b1ef1-19fb-49d3-878b-b6b85f333998","Type":"ContainerStarted","Data":"fc27a012936ab47d9bebc2b92aaf89d8a9a5f4683b0297474ef427f75516a037"}
Jul 30 17:01:52.542614 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:52.542586    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6448464666-hls4h" event={"ID":"266b5284-fb72-46c4-a694-37535e997cd3","Type":"ContainerStarted","Data":"0e5f3dd623c0f2cee3faffe0ac4173ffaf4dd856bf11811e05ea40d43638d106"}
Jul 30 17:01:52.542733 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:52.542641    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6448464666-hls4h" event={"ID":"266b5284-fb72-46c4-a694-37535e997cd3","Type":"ContainerStarted","Data":"24cddd49e17a15a9b12fbb877b4ddacd3e8b4bbc10c5a949e47a91454fa6d7e6"}
Jul 30 17:01:52.544350 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:52.544323    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-dlr24" event={"ID":"411dc2d0-e97f-4574-abe7-1e06f22653f9","Type":"ContainerStarted","Data":"f166f7a097ba134a8e559e75b8d94049ce578ab27a8487f80b9b489ffbb67306"}
Jul 30 17:01:52.545110 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:52.545080    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-dlr24"
Jul 30 17:01:52.547242 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:52.547217    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7675687bb6-jkh86" event={"ID":"f4fb739f-b8d6-494d-987d-cf9def14829e","Type":"ContainerStarted","Data":"4e91414c4cbd4abfe82eef78734ad0f33cfcf99390697b0fbd2782ebb5947b41"}
Jul 30 17:01:52.553261 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:52.553222    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-dlr24"
Jul 30 17:01:52.557809 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:52.557666    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7545856648-6x6pd" podStartSLOduration=10.638147159 podStartE2EDuration="16.557648244s" totalImagesPullingTime="5.919501085s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:01:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:01:52.557143272 +0000 UTC m=+87.044510675" watchObservedRunningTime="2026-07-30 17:01:52.557648244 +0000 UTC m=+87.045015646"
Jul 30 17:01:52.597896 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:52.597715    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-8447bbff85-pspxs" podStartSLOduration=3.828559485 podStartE2EDuration="9.597687619s" totalImagesPullingTime="5.769128134s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:01:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:01:52.593920154 +0000 UTC m=+87.081287548" watchObservedRunningTime="2026-07-30 17:01:52.597687619 +0000 UTC m=+87.085055015"
Jul 30 17:01:52.599444 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:52.599396    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7675687bb6-jkh86" podStartSLOduration=11.999787008 podStartE2EDuration="18.599379278s" totalImagesPullingTime="6.59959227s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:01:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:01:52.57565264 +0000 UTC m=+87.063020073" watchObservedRunningTime="2026-07-30 17:01:52.599379278 +0000 UTC m=+87.086746682"
Jul 30 17:01:52.620848 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:52.620792    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-dlr24" podStartSLOduration=10.702027699 podStartE2EDuration="16.62077655s" totalImagesPullingTime="5.918748851s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:01:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:01:52.619846851 +0000 UTC m=+87.107214253" watchObservedRunningTime="2026-07-30 17:01:52.62077655 +0000 UTC m=+87.108143951"
Jul 30 17:01:53.555944 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:53.555893    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6448464666-hls4h" event={"ID":"266b5284-fb72-46c4-a694-37535e997cd3","Type":"ContainerStarted","Data":"cb3c2edaa25fb78d607a647d8c6c272fd8b777ad160cc0ed5381129bdedea2a7"}
Jul 30 17:01:53.557147 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:53.556478    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6448464666-hls4h"
Jul 30 17:01:53.563019 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:53.562992    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6448464666-hls4h"
Jul 30 17:01:53.587260 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:53.587200    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6448464666-hls4h" podStartSLOduration=5.286947543 podStartE2EDuration="20.587178403s" totalImagesPullingTime="15.30023086s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:01:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:01:53.585955895 +0000 UTC m=+88.073323319" watchObservedRunningTime="2026-07-30 17:01:53.587178403 +0000 UTC m=+88.074545804"
Jul 30 17:01:54.025357 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:54.025262    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:54.126466 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:54.126430    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:54.252612 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:54.252524    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8447bbff85-pspxs"
Jul 30 17:01:54.252969 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:54.252920    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8447bbff85-pspxs"
Jul 30 17:01:54.258559 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:54.258535    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8447bbff85-pspxs"
Jul 30 17:01:54.565123 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:54.565094    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8447bbff85-pspxs"
Jul 30 17:01:54.686197 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:54.686156    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7675687bb6-jkh86"
Jul 30 17:01:54.882269 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:54.882186    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:54.983148 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:54.983112    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:55.143353 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:55.143270    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:55.202176 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:55.202135    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:55.963356 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:55.963326    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:56.030697 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:56.030667    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:56.386649 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:56.386611    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7545856648-6x6pd"
Jul 30 17:01:56.386828 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:56.386687    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7545856648-6x6pd"
Jul 30 17:01:58.539460 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:58.539423    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:01:58.590786 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:01:58.590756    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:02:09.918888 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:09.918854    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:02:09.973919 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:09.973883    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:02:16.393065 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:16.393027    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7545856648-6x6pd"
Jul 30 17:02:16.397401 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:16.397374    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7545856648-6x6pd"
Jul 30 17:02:17.580105 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:17.580057    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7675687bb6-jkh86" podUID="f4fb739f-b8d6-494d-987d-cf9def14829e" containerName="console" containerID="cri-o://4e91414c4cbd4abfe82eef78734ad0f33cfcf99390697b0fbd2782ebb5947b41" gracePeriod=15
Jul 30 17:02:17.864254 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:17.864228    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7675687bb6-jkh86"
Jul 30 17:02:17.908675 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:17.908642    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8sjf6\" (UniqueName: \"kubernetes.io/projected/f4fb739f-b8d6-494d-987d-cf9def14829e-kube-api-access-8sjf6\") pod \"f4fb739f-b8d6-494d-987d-cf9def14829e\" (UID: \"f4fb739f-b8d6-494d-987d-cf9def14829e\") "
Jul 30 17:02:17.908675 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:17.908681    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f4fb739f-b8d6-494d-987d-cf9def14829e-console-config\") pod \"f4fb739f-b8d6-494d-987d-cf9def14829e\" (UID: \"f4fb739f-b8d6-494d-987d-cf9def14829e\") "
Jul 30 17:02:17.908879 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:17.908707    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f4fb739f-b8d6-494d-987d-cf9def14829e-oauth-serving-cert\") pod \"f4fb739f-b8d6-494d-987d-cf9def14829e\" (UID: \"f4fb739f-b8d6-494d-987d-cf9def14829e\") "
Jul 30 17:02:17.908879 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:17.908748    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4fb739f-b8d6-494d-987d-cf9def14829e-trusted-ca-bundle\") pod \"f4fb739f-b8d6-494d-987d-cf9def14829e\" (UID: \"f4fb739f-b8d6-494d-987d-cf9def14829e\") "
Jul 30 17:02:17.908879 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:17.908765    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f4fb739f-b8d6-494d-987d-cf9def14829e-console-oauth-config\") pod \"f4fb739f-b8d6-494d-987d-cf9def14829e\" (UID: \"f4fb739f-b8d6-494d-987d-cf9def14829e\") "
Jul 30 17:02:17.908879 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:17.908807    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f4fb739f-b8d6-494d-987d-cf9def14829e-service-ca\") pod \"f4fb739f-b8d6-494d-987d-cf9def14829e\" (UID: \"f4fb739f-b8d6-494d-987d-cf9def14829e\") "
Jul 30 17:02:17.909088 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:17.908978    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f4fb739f-b8d6-494d-987d-cf9def14829e-console-serving-cert\") pod \"f4fb739f-b8d6-494d-987d-cf9def14829e\" (UID: \"f4fb739f-b8d6-494d-987d-cf9def14829e\") "
Jul 30 17:02:17.909169 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:17.909149    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4fb739f-b8d6-494d-987d-cf9def14829e-console-config" (OuterVolumeSpecName: "console-config") pod "f4fb739f-b8d6-494d-987d-cf9def14829e" (UID: "f4fb739f-b8d6-494d-987d-cf9def14829e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:02:17.909253 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:17.909195    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4fb739f-b8d6-494d-987d-cf9def14829e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f4fb739f-b8d6-494d-987d-cf9def14829e" (UID: "f4fb739f-b8d6-494d-987d-cf9def14829e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:02:17.909253 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:17.909229    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4fb739f-b8d6-494d-987d-cf9def14829e-service-ca" (OuterVolumeSpecName: "service-ca") pod "f4fb739f-b8d6-494d-987d-cf9def14829e" (UID: "f4fb739f-b8d6-494d-987d-cf9def14829e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:02:17.909368 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:17.909248    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4fb739f-b8d6-494d-987d-cf9def14829e-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "f4fb739f-b8d6-494d-987d-cf9def14829e" (UID: "f4fb739f-b8d6-494d-987d-cf9def14829e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:02:17.909368 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:17.909317    2555 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f4fb739f-b8d6-494d-987d-cf9def14829e-console-config\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:02:17.909368 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:17.909330    2555 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f4fb739f-b8d6-494d-987d-cf9def14829e-oauth-serving-cert\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:02:17.909368 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:17.909340    2555 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4fb739f-b8d6-494d-987d-cf9def14829e-trusted-ca-bundle\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:02:17.909368 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:17.909352    2555 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f4fb739f-b8d6-494d-987d-cf9def14829e-service-ca\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:02:17.911084 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:17.911057    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4fb739f-b8d6-494d-987d-cf9def14829e-kube-api-access-8sjf6" (OuterVolumeSpecName: "kube-api-access-8sjf6") pod "f4fb739f-b8d6-494d-987d-cf9def14829e" (UID: "f4fb739f-b8d6-494d-987d-cf9def14829e"). InnerVolumeSpecName "kube-api-access-8sjf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 17:02:17.911187 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:17.911100    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4fb739f-b8d6-494d-987d-cf9def14829e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f4fb739f-b8d6-494d-987d-cf9def14829e" (UID: "f4fb739f-b8d6-494d-987d-cf9def14829e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:02:17.911227 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:17.911186    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4fb739f-b8d6-494d-987d-cf9def14829e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f4fb739f-b8d6-494d-987d-cf9def14829e" (UID: "f4fb739f-b8d6-494d-987d-cf9def14829e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:02:18.010463 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:18.010431    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sjf6\" (UniqueName: \"kubernetes.io/projected/f4fb739f-b8d6-494d-987d-cf9def14829e-kube-api-access-8sjf6\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:02:18.010463 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:18.010461    2555 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f4fb739f-b8d6-494d-987d-cf9def14829e-console-oauth-config\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:02:18.010463 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:18.010470    2555 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f4fb739f-b8d6-494d-987d-cf9def14829e-console-serving-cert\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:02:18.520327 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:18.520298    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-qvgrv"
Jul 30 17:02:18.651585 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:18.651548    2555 generic.go:356] "Generic (PLEG): container finished" podID="f4fb739f-b8d6-494d-987d-cf9def14829e" containerID="4e91414c4cbd4abfe82eef78734ad0f33cfcf99390697b0fbd2782ebb5947b41" exitCode=0
Jul 30 17:02:18.651585 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:18.651591    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7675687bb6-jkh86"
Jul 30 17:02:18.652062 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:18.651643    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7675687bb6-jkh86" event={"ID":"f4fb739f-b8d6-494d-987d-cf9def14829e","Type":"ContainerDied","Data":"4e91414c4cbd4abfe82eef78734ad0f33cfcf99390697b0fbd2782ebb5947b41"}
Jul 30 17:02:18.652062 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:18.651691    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7675687bb6-jkh86" event={"ID":"f4fb739f-b8d6-494d-987d-cf9def14829e","Type":"ContainerDied","Data":"e75554b666bc061b3ce5d8d4a146a7550fbe5c1dba6d229111d3654491fb5fe1"}
Jul 30 17:02:18.652062 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:18.651712    2555 scope.go:117] "RemoveContainer" containerID="4e91414c4cbd4abfe82eef78734ad0f33cfcf99390697b0fbd2782ebb5947b41"
Jul 30 17:02:18.659928 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:18.659909    2555 scope.go:117] "RemoveContainer" containerID="4e91414c4cbd4abfe82eef78734ad0f33cfcf99390697b0fbd2782ebb5947b41"
Jul 30 17:02:18.660211 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:02:18.660193    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e91414c4cbd4abfe82eef78734ad0f33cfcf99390697b0fbd2782ebb5947b41\": container with ID starting with 4e91414c4cbd4abfe82eef78734ad0f33cfcf99390697b0fbd2782ebb5947b41 not found: ID does not exist" containerID="4e91414c4cbd4abfe82eef78734ad0f33cfcf99390697b0fbd2782ebb5947b41"
Jul 30 17:02:18.660267 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:18.660222    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e91414c4cbd4abfe82eef78734ad0f33cfcf99390697b0fbd2782ebb5947b41"} err="failed to get container status \"4e91414c4cbd4abfe82eef78734ad0f33cfcf99390697b0fbd2782ebb5947b41\": rpc error: code = NotFound desc = could not find container \"4e91414c4cbd4abfe82eef78734ad0f33cfcf99390697b0fbd2782ebb5947b41\": container with ID starting with 4e91414c4cbd4abfe82eef78734ad0f33cfcf99390697b0fbd2782ebb5947b41 not found: ID does not exist"
Jul 30 17:02:18.669519 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:18.669496    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7675687bb6-jkh86"]
Jul 30 17:02:18.672703 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:18.672679    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7675687bb6-jkh86"]
Jul 30 17:02:20.060147 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:20.060114    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4fb739f-b8d6-494d-987d-cf9def14829e" path="/var/lib/kubelet/pods/f4fb739f-b8d6-494d-987d-cf9def14829e/volumes"
Jul 30 17:02:28.788212 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:28.788178    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:02:52.097943 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:52.097908    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:02:52.197640 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:52.197607    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:02:54.366770 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:54.366739    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 30 17:02:54.367162 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:54.367025    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4fb739f-b8d6-494d-987d-cf9def14829e" containerName="console"
Jul 30 17:02:54.367162 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:54.367036    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4fb739f-b8d6-494d-987d-cf9def14829e" containerName="console"
Jul 30 17:02:54.367162 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:54.367082    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4fb739f-b8d6-494d-987d-cf9def14829e" containerName="console"
Jul 30 17:02:54.369900 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:54.369883    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:02:54.372855 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:54.372821    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 30 17:02:54.372855 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:54.372844    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-62qft\""
Jul 30 17:02:54.373047 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:54.372823    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 30 17:02:54.373284 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:54.373268    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 30 17:02:54.373284 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:54.373280    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 30 17:02:54.373398 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:54.373268    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 30 17:02:54.373606 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:54.373593    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 30 17:02:54.373656 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:54.373593    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 30 17:02:54.374178 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:54.374147    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 30 17:02:54.382435 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:54.382405    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 30 17:02:54.385317 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:54.385297    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 30 17:02:54.501631 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:54.501590    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c66554ef-e555-43c5-b2eb-cdaa50f6e3d4-config-volume\") pod \"alertmanager-main-0\" (UID: \"c66554ef-e555-43c5-b2eb-cdaa50f6e3d4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:02:54.501631 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:54.501630    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c66554ef-e555-43c5-b2eb-cdaa50f6e3d4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c66554ef-e555-43c5-b2eb-cdaa50f6e3d4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:02:54.501874 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:54.501649    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c66554ef-e555-43c5-b2eb-cdaa50f6e3d4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c66554ef-e555-43c5-b2eb-cdaa50f6e3d4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:02:54.501874 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:54.501677    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c66554ef-e555-43c5-b2eb-cdaa50f6e3d4-web-config\") pod \"alertmanager-main-0\" (UID: \"c66554ef-e555-43c5-b2eb-cdaa50f6e3d4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:02:54.501874 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:54.501762    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c66554ef-e555-43c5-b2eb-cdaa50f6e3d4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c66554ef-e555-43c5-b2eb-cdaa50f6e3d4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:02:54.501874 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:54.501800    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c66554ef-e555-43c5-b2eb-cdaa50f6e3d4-config-out\") pod \"alertmanager-main-0\" (UID: \"c66554ef-e555-43c5-b2eb-cdaa50f6e3d4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:02:54.501874 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:54.501865    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c66554ef-e555-43c5-b2eb-cdaa50f6e3d4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c66554ef-e555-43c5-b2eb-cdaa50f6e3d4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:02:54.502077 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:54.501904    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c66554ef-e555-43c5-b2eb-cdaa50f6e3d4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c66554ef-e555-43c5-b2eb-cdaa50f6e3d4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:02:54.502077 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:54.501933    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c66554ef-e555-43c5-b2eb-cdaa50f6e3d4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c66554ef-e555-43c5-b2eb-cdaa50f6e3d4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:02:54.502077 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:54.501953    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c66554ef-e555-43c5-b2eb-cdaa50f6e3d4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c66554ef-e555-43c5-b2eb-cdaa50f6e3d4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:02:54.502077 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:54.501979    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krxwd\" (UniqueName: \"kubernetes.io/projected/c66554ef-e555-43c5-b2eb-cdaa50f6e3d4-kube-api-access-krxwd\") pod \"alertmanager-main-0\" (UID: \"c66554ef-e555-43c5-b2eb-cdaa50f6e3d4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:02:54.502077 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:54.502057    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c66554ef-e555-43c5-b2eb-cdaa50f6e3d4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c66554ef-e555-43c5-b2eb-cdaa50f6e3d4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:02:54.502229 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:54.502088    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c66554ef-e555-43c5-b2eb-cdaa50f6e3d4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c66554ef-e555-43c5-b2eb-cdaa50f6e3d4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:02:54.603314 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:54.603284    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c66554ef-e555-43c5-b2eb-cdaa50f6e3d4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c66554ef-e555-43c5-b2eb-cdaa50f6e3d4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:02:54.603510 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:54.603329    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c66554ef-e555-43c5-b2eb-cdaa50f6e3d4-config-volume\") pod \"alertmanager-main-0\" (UID: \"c66554ef-e555-43c5-b2eb-cdaa50f6e3d4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:02:54.603510 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:54.603367    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c66554ef-e555-43c5-b2eb-cdaa50f6e3d4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c66554ef-e555-43c5-b2eb-cdaa50f6e3d4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:02:54.603615 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:54.603545    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c66554ef-e555-43c5-b2eb-cdaa50f6e3d4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c66554ef-e555-43c5-b2eb-cdaa50f6e3d4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:02:54.603615 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:54.603594    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c66554ef-e555-43c5-b2eb-cdaa50f6e3d4-web-config\") pod \"alertmanager-main-0\" (UID: \"c66554ef-e555-43c5-b2eb-cdaa50f6e3d4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:02:54.603719 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:54.603629    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c66554ef-e555-43c5-b2eb-cdaa50f6e3d4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c66554ef-e555-43c5-b2eb-cdaa50f6e3d4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:02:54.603719 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:54.603659    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c66554ef-e555-43c5-b2eb-cdaa50f6e3d4-config-out\") pod \"alertmanager-main-0\" (UID: \"c66554ef-e555-43c5-b2eb-cdaa50f6e3d4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:02:54.603719 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:54.603706    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c66554ef-e555-43c5-b2eb-cdaa50f6e3d4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c66554ef-e555-43c5-b2eb-cdaa50f6e3d4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:02:54.603868 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:54.603747    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c66554ef-e555-43c5-b2eb-cdaa50f6e3d4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c66554ef-e555-43c5-b2eb-cdaa50f6e3d4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:02:54.603868 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:54.603793    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c66554ef-e555-43c5-b2eb-cdaa50f6e3d4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c66554ef-e555-43c5-b2eb-cdaa50f6e3d4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:02:54.603868 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:54.603818    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c66554ef-e555-43c5-b2eb-cdaa50f6e3d4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c66554ef-e555-43c5-b2eb-cdaa50f6e3d4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:02:54.603868 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:54.603842    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krxwd\" (UniqueName: \"kubernetes.io/projected/c66554ef-e555-43c5-b2eb-cdaa50f6e3d4-kube-api-access-krxwd\") pod \"alertmanager-main-0\" (UID: \"c66554ef-e555-43c5-b2eb-cdaa50f6e3d4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:02:54.604098 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:54.603887    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c66554ef-e555-43c5-b2eb-cdaa50f6e3d4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c66554ef-e555-43c5-b2eb-cdaa50f6e3d4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:02:54.604470 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:54.604342    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c66554ef-e555-43c5-b2eb-cdaa50f6e3d4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c66554ef-e555-43c5-b2eb-cdaa50f6e3d4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:02:54.604470 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:54.604392    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c66554ef-e555-43c5-b2eb-cdaa50f6e3d4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c66554ef-e555-43c5-b2eb-cdaa50f6e3d4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:02:54.605095 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:54.604729    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c66554ef-e555-43c5-b2eb-cdaa50f6e3d4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c66554ef-e555-43c5-b2eb-cdaa50f6e3d4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:02:54.606475 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:54.606431    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c66554ef-e555-43c5-b2eb-cdaa50f6e3d4-config-volume\") pod \"alertmanager-main-0\" (UID: \"c66554ef-e555-43c5-b2eb-cdaa50f6e3d4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:02:54.606576 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:54.606554    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c66554ef-e555-43c5-b2eb-cdaa50f6e3d4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c66554ef-e555-43c5-b2eb-cdaa50f6e3d4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:02:54.606639 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:54.606554    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c66554ef-e555-43c5-b2eb-cdaa50f6e3d4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c66554ef-e555-43c5-b2eb-cdaa50f6e3d4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:02:54.606831 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:54.606783    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c66554ef-e555-43c5-b2eb-cdaa50f6e3d4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c66554ef-e555-43c5-b2eb-cdaa50f6e3d4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:02:54.606951 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:54.606939    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c66554ef-e555-43c5-b2eb-cdaa50f6e3d4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c66554ef-e555-43c5-b2eb-cdaa50f6e3d4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:02:54.607350 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:54.607328    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c66554ef-e555-43c5-b2eb-cdaa50f6e3d4-web-config\") pod \"alertmanager-main-0\" (UID: \"c66554ef-e555-43c5-b2eb-cdaa50f6e3d4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:02:54.607425 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:54.607362    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c66554ef-e555-43c5-b2eb-cdaa50f6e3d4-config-out\") pod \"alertmanager-main-0\" (UID: \"c66554ef-e555-43c5-b2eb-cdaa50f6e3d4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:02:54.607784 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:54.607764    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c66554ef-e555-43c5-b2eb-cdaa50f6e3d4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c66554ef-e555-43c5-b2eb-cdaa50f6e3d4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:02:54.608517 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:54.608501    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c66554ef-e555-43c5-b2eb-cdaa50f6e3d4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c66554ef-e555-43c5-b2eb-cdaa50f6e3d4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:02:54.614768 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:54.614741    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krxwd\" (UniqueName: \"kubernetes.io/projected/c66554ef-e555-43c5-b2eb-cdaa50f6e3d4-kube-api-access-krxwd\") pod \"alertmanager-main-0\" (UID: \"c66554ef-e555-43c5-b2eb-cdaa50f6e3d4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:02:54.683436 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:54.683358    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:02:54.814818 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:54.814789    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 30 17:02:54.818115 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:02:54.818087    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc66554ef_e555_43c5_b2eb_cdaa50f6e3d4.slice/crio-0c054e7e0c86022b15da2edf12d0f71a9b8da0c5cb18ef9d4ae159e252811f63 WatchSource:0}: Error finding container 0c054e7e0c86022b15da2edf12d0f71a9b8da0c5cb18ef9d4ae159e252811f63: Status 404 returned error can't find the container with id 0c054e7e0c86022b15da2edf12d0f71a9b8da0c5cb18ef9d4ae159e252811f63
Jul 30 17:02:55.789997 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:55.789966    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c66554ef-e555-43c5-b2eb-cdaa50f6e3d4","Type":"ContainerStarted","Data":"0c054e7e0c86022b15da2edf12d0f71a9b8da0c5cb18ef9d4ae159e252811f63"}
Jul 30 17:02:56.163569 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:56.163528    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7679956457-564g9"]
Jul 30 17:02:56.166241 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:56.166221    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7679956457-564g9"
Jul 30 17:02:56.169448 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:56.169020    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-fqldh\""
Jul 30 17:02:56.169448 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:56.169303    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 30 17:02:56.170304 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:56.169765    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 30 17:02:56.170304 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:56.170153    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 30 17:02:56.171078 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:56.170635    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 30 17:02:56.171204 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:56.171192    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 30 17:02:56.174258 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:56.174233    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 30 17:02:56.178049 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:56.178029    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7679956457-564g9"]
Jul 30 17:02:56.216828 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:56.216793    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4e9daaaf-67dc-4b6c-bae1-2e2d092d4e5d-metrics-client-ca\") pod \"telemeter-client-7679956457-564g9\" (UID: \"4e9daaaf-67dc-4b6c-bae1-2e2d092d4e5d\") " pod="openshift-monitoring/telemeter-client-7679956457-564g9"
Jul 30 17:02:56.217041 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:56.216837    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e9daaaf-67dc-4b6c-bae1-2e2d092d4e5d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7679956457-564g9\" (UID: \"4e9daaaf-67dc-4b6c-bae1-2e2d092d4e5d\") " pod="openshift-monitoring/telemeter-client-7679956457-564g9"
Jul 30 17:02:56.217041 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:56.216901    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/4e9daaaf-67dc-4b6c-bae1-2e2d092d4e5d-secret-telemeter-client\") pod \"telemeter-client-7679956457-564g9\" (UID: \"4e9daaaf-67dc-4b6c-bae1-2e2d092d4e5d\") " pod="openshift-monitoring/telemeter-client-7679956457-564g9"
Jul 30 17:02:56.217041 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:56.216923    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/4e9daaaf-67dc-4b6c-bae1-2e2d092d4e5d-federate-client-tls\") pod \"telemeter-client-7679956457-564g9\" (UID: \"4e9daaaf-67dc-4b6c-bae1-2e2d092d4e5d\") " pod="openshift-monitoring/telemeter-client-7679956457-564g9"
Jul 30 17:02:56.217041 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:56.216981    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/4e9daaaf-67dc-4b6c-bae1-2e2d092d4e5d-telemeter-client-tls\") pod \"telemeter-client-7679956457-564g9\" (UID: \"4e9daaaf-67dc-4b6c-bae1-2e2d092d4e5d\") " pod="openshift-monitoring/telemeter-client-7679956457-564g9"
Jul 30 17:02:56.217236 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:56.217047    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78rlz\" (UniqueName: \"kubernetes.io/projected/4e9daaaf-67dc-4b6c-bae1-2e2d092d4e5d-kube-api-access-78rlz\") pod \"telemeter-client-7679956457-564g9\" (UID: \"4e9daaaf-67dc-4b6c-bae1-2e2d092d4e5d\") " pod="openshift-monitoring/telemeter-client-7679956457-564g9"
Jul 30 17:02:56.217236 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:56.217074    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4e9daaaf-67dc-4b6c-bae1-2e2d092d4e5d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7679956457-564g9\" (UID: \"4e9daaaf-67dc-4b6c-bae1-2e2d092d4e5d\") " pod="openshift-monitoring/telemeter-client-7679956457-564g9"
Jul 30 17:02:56.217236 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:56.217140    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e9daaaf-67dc-4b6c-bae1-2e2d092d4e5d-serving-certs-ca-bundle\") pod \"telemeter-client-7679956457-564g9\" (UID: \"4e9daaaf-67dc-4b6c-bae1-2e2d092d4e5d\") " pod="openshift-monitoring/telemeter-client-7679956457-564g9"
Jul 30 17:02:56.317774 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:56.317737    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/4e9daaaf-67dc-4b6c-bae1-2e2d092d4e5d-secret-telemeter-client\") pod \"telemeter-client-7679956457-564g9\" (UID: \"4e9daaaf-67dc-4b6c-bae1-2e2d092d4e5d\") " pod="openshift-monitoring/telemeter-client-7679956457-564g9"
Jul 30 17:02:56.317774 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:56.317770    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/4e9daaaf-67dc-4b6c-bae1-2e2d092d4e5d-federate-client-tls\") pod \"telemeter-client-7679956457-564g9\" (UID: \"4e9daaaf-67dc-4b6c-bae1-2e2d092d4e5d\") " pod="openshift-monitoring/telemeter-client-7679956457-564g9"
Jul 30 17:02:56.318044 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:56.317798    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/4e9daaaf-67dc-4b6c-bae1-2e2d092d4e5d-telemeter-client-tls\") pod \"telemeter-client-7679956457-564g9\" (UID: \"4e9daaaf-67dc-4b6c-bae1-2e2d092d4e5d\") " pod="openshift-monitoring/telemeter-client-7679956457-564g9"
Jul 30 17:02:56.318044 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:56.317826    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78rlz\" (UniqueName: \"kubernetes.io/projected/4e9daaaf-67dc-4b6c-bae1-2e2d092d4e5d-kube-api-access-78rlz\") pod \"telemeter-client-7679956457-564g9\" (UID: \"4e9daaaf-67dc-4b6c-bae1-2e2d092d4e5d\") " pod="openshift-monitoring/telemeter-client-7679956457-564g9"
Jul 30 17:02:56.318044 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:56.317853    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4e9daaaf-67dc-4b6c-bae1-2e2d092d4e5d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7679956457-564g9\" (UID: \"4e9daaaf-67dc-4b6c-bae1-2e2d092d4e5d\") " pod="openshift-monitoring/telemeter-client-7679956457-564g9"
Jul 30 17:02:56.318044 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:56.317891    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e9daaaf-67dc-4b6c-bae1-2e2d092d4e5d-serving-certs-ca-bundle\") pod \"telemeter-client-7679956457-564g9\" (UID: \"4e9daaaf-67dc-4b6c-bae1-2e2d092d4e5d\") " pod="openshift-monitoring/telemeter-client-7679956457-564g9"
Jul 30 17:02:56.318044 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:56.317938    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4e9daaaf-67dc-4b6c-bae1-2e2d092d4e5d-metrics-client-ca\") pod \"telemeter-client-7679956457-564g9\" (UID: \"4e9daaaf-67dc-4b6c-bae1-2e2d092d4e5d\") " pod="openshift-monitoring/telemeter-client-7679956457-564g9"
Jul 30 17:02:56.318044 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:56.317968    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e9daaaf-67dc-4b6c-bae1-2e2d092d4e5d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7679956457-564g9\" (UID: \"4e9daaaf-67dc-4b6c-bae1-2e2d092d4e5d\") " pod="openshift-monitoring/telemeter-client-7679956457-564g9"
Jul 30 17:02:56.318670 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:56.318637    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e9daaaf-67dc-4b6c-bae1-2e2d092d4e5d-serving-certs-ca-bundle\") pod \"telemeter-client-7679956457-564g9\" (UID: \"4e9daaaf-67dc-4b6c-bae1-2e2d092d4e5d\") " pod="openshift-monitoring/telemeter-client-7679956457-564g9"
Jul 30 17:02:56.318793 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:56.318775    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4e9daaaf-67dc-4b6c-bae1-2e2d092d4e5d-metrics-client-ca\") pod \"telemeter-client-7679956457-564g9\" (UID: \"4e9daaaf-67dc-4b6c-bae1-2e2d092d4e5d\") " pod="openshift-monitoring/telemeter-client-7679956457-564g9"
Jul 30 17:02:56.319424 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:56.319404    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e9daaaf-67dc-4b6c-bae1-2e2d092d4e5d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7679956457-564g9\" (UID: \"4e9daaaf-67dc-4b6c-bae1-2e2d092d4e5d\") " pod="openshift-monitoring/telemeter-client-7679956457-564g9"
Jul 30 17:02:56.320708 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:56.320675    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/4e9daaaf-67dc-4b6c-bae1-2e2d092d4e5d-federate-client-tls\") pod \"telemeter-client-7679956457-564g9\" (UID: \"4e9daaaf-67dc-4b6c-bae1-2e2d092d4e5d\") " pod="openshift-monitoring/telemeter-client-7679956457-564g9"
Jul 30 17:02:56.320822 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:56.320779    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/4e9daaaf-67dc-4b6c-bae1-2e2d092d4e5d-telemeter-client-tls\") pod \"telemeter-client-7679956457-564g9\" (UID: \"4e9daaaf-67dc-4b6c-bae1-2e2d092d4e5d\") " pod="openshift-monitoring/telemeter-client-7679956457-564g9"
Jul 30 17:02:56.320894 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:56.320852    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4e9daaaf-67dc-4b6c-bae1-2e2d092d4e5d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7679956457-564g9\" (UID: \"4e9daaaf-67dc-4b6c-bae1-2e2d092d4e5d\") " pod="openshift-monitoring/telemeter-client-7679956457-564g9"
Jul 30 17:02:56.321002 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:56.320977    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/4e9daaaf-67dc-4b6c-bae1-2e2d092d4e5d-secret-telemeter-client\") pod \"telemeter-client-7679956457-564g9\" (UID: \"4e9daaaf-67dc-4b6c-bae1-2e2d092d4e5d\") " pod="openshift-monitoring/telemeter-client-7679956457-564g9"
Jul 30 17:02:56.330081 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:56.330039    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78rlz\" (UniqueName: \"kubernetes.io/projected/4e9daaaf-67dc-4b6c-bae1-2e2d092d4e5d-kube-api-access-78rlz\") pod \"telemeter-client-7679956457-564g9\" (UID: \"4e9daaaf-67dc-4b6c-bae1-2e2d092d4e5d\") " pod="openshift-monitoring/telemeter-client-7679956457-564g9"
Jul 30 17:02:56.478532 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:56.478449    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7679956457-564g9"
Jul 30 17:02:56.626951 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:56.626917    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7679956457-564g9"]
Jul 30 17:02:56.631991 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:02:56.631965    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e9daaaf_67dc_4b6c_bae1_2e2d092d4e5d.slice/crio-a74b06afab492d6fe8e3ec4385f29f657566b4ef1d455e0936391be18ae8f743 WatchSource:0}: Error finding container a74b06afab492d6fe8e3ec4385f29f657566b4ef1d455e0936391be18ae8f743: Status 404 returned error can't find the container with id a74b06afab492d6fe8e3ec4385f29f657566b4ef1d455e0936391be18ae8f743
Jul 30 17:02:56.794661 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:56.794614    2555 generic.go:356] "Generic (PLEG): container finished" podID="c66554ef-e555-43c5-b2eb-cdaa50f6e3d4" containerID="cb76a9df1295cb4ea9daf8a35f599055ab492f0c975ea2a221b45c2837ef134f" exitCode=0
Jul 30 17:02:56.795132 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:56.794702    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c66554ef-e555-43c5-b2eb-cdaa50f6e3d4","Type":"ContainerDied","Data":"cb76a9df1295cb4ea9daf8a35f599055ab492f0c975ea2a221b45c2837ef134f"}
Jul 30 17:02:56.795855 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:56.795829    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7679956457-564g9" event={"ID":"4e9daaaf-67dc-4b6c-bae1-2e2d092d4e5d","Type":"ContainerStarted","Data":"a74b06afab492d6fe8e3ec4385f29f657566b4ef1d455e0936391be18ae8f743"}
Jul 30 17:02:58.806803 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:58.806773    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c66554ef-e555-43c5-b2eb-cdaa50f6e3d4","Type":"ContainerStarted","Data":"1b15724a543c20792328540ac4f478ff73b3b77486ec45d61e421a09f194441f"}
Jul 30 17:02:58.807201 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:58.806842    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c66554ef-e555-43c5-b2eb-cdaa50f6e3d4","Type":"ContainerStarted","Data":"47c4a8321d54fd34a8b546e12bf3fb40461c80871df133c95c7b7de4716a0ff7"}
Jul 30 17:02:58.808144 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:58.808123    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7679956457-564g9" event={"ID":"4e9daaaf-67dc-4b6c-bae1-2e2d092d4e5d","Type":"ContainerStarted","Data":"c30830b6dd366424627b94e671d2ddc39dc64a144ac5605f38aef68cfcae9cea"}
Jul 30 17:02:58.808241 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:58.808171    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7679956457-564g9" event={"ID":"4e9daaaf-67dc-4b6c-bae1-2e2d092d4e5d","Type":"ContainerStarted","Data":"d7edc6c17a64c625730197968dfa242465ae48f06b4a289d59e4ac69dfb17caa"}
Jul 30 17:02:59.815302 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:59.815266    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c66554ef-e555-43c5-b2eb-cdaa50f6e3d4","Type":"ContainerStarted","Data":"109d06f4fe9d4bf100dc4d102a5004ef41c942eccc6c32274e65afa91f308100"}
Jul 30 17:02:59.815753 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:59.815334    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c66554ef-e555-43c5-b2eb-cdaa50f6e3d4","Type":"ContainerStarted","Data":"4f08d44f3cdc03993cf694c3736cfb97dcffaf7d678a7ebfcb843afb1aee3d94"}
Jul 30 17:02:59.815753 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:59.815365    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c66554ef-e555-43c5-b2eb-cdaa50f6e3d4","Type":"ContainerStarted","Data":"c7e45ff5f632db8cb3597679c17f7829cd32211c6fd929f746463dc133813dc8"}
Jul 30 17:02:59.815753 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:59.815403    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c66554ef-e555-43c5-b2eb-cdaa50f6e3d4","Type":"ContainerStarted","Data":"02ecdcb6c450f69b2e3cb36ca402ec240482b54d09077caec9170baeb8db41fd"}
Jul 30 17:02:59.817371 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:59.817349    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7679956457-564g9" event={"ID":"4e9daaaf-67dc-4b6c-bae1-2e2d092d4e5d","Type":"ContainerStarted","Data":"8dbc91786b16e5148523c433c11ce61a5d9a2598aa44ac3af1209f7397efe0e5"}
Jul 30 17:02:59.844528 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:59.844472    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.973305801 podStartE2EDuration="5.844452786s" totalImagesPullingTime="2.855542971s" totalInitContainerRuntime="15.604014ms" isStatefulPod=false podCreationTimestamp="2026-07-30 17:02:54 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:02:59.844075566 +0000 UTC m=+154.331442968" watchObservedRunningTime="2026-07-30 17:02:59.844452786 +0000 UTC m=+154.331820188"
Jul 30 17:02:59.871652 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:02:59.871604    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7679956457-564g9" podStartSLOduration=1.863565877 podStartE2EDuration="3.871590861s" totalImagesPullingTime="2.008024984s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:02:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:02:59.869878231 +0000 UTC m=+154.357245631" watchObservedRunningTime="2026-07-30 17:02:59.871590861 +0000 UTC m=+154.358958262"
Jul 30 17:03:00.635835 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:03:00.635802    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-59855b657f-8qznz"]
Jul 30 17:03:00.638202 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:03:00.638186    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59855b657f-8qznz"
Jul 30 17:03:00.649547 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:03:00.649523    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59855b657f-8qznz"]
Jul 30 17:03:00.651427 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:03:00.651403    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c494027f-c3f1-4e9f-ba91-7f2c30580969-console-config\") pod \"console-59855b657f-8qznz\" (UID: \"c494027f-c3f1-4e9f-ba91-7f2c30580969\") " pod="openshift-console/console-59855b657f-8qznz"
Jul 30 17:03:00.651513 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:03:00.651438    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vczc\" (UniqueName: \"kubernetes.io/projected/c494027f-c3f1-4e9f-ba91-7f2c30580969-kube-api-access-8vczc\") pod \"console-59855b657f-8qznz\" (UID: \"c494027f-c3f1-4e9f-ba91-7f2c30580969\") " pod="openshift-console/console-59855b657f-8qznz"
Jul 30 17:03:00.651513 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:03:00.651461    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c494027f-c3f1-4e9f-ba91-7f2c30580969-service-ca\") pod \"console-59855b657f-8qznz\" (UID: \"c494027f-c3f1-4e9f-ba91-7f2c30580969\") " pod="openshift-console/console-59855b657f-8qznz"
Jul 30 17:03:00.651599 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:03:00.651574    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c494027f-c3f1-4e9f-ba91-7f2c30580969-console-serving-cert\") pod \"console-59855b657f-8qznz\" (UID: \"c494027f-c3f1-4e9f-ba91-7f2c30580969\") " pod="openshift-console/console-59855b657f-8qznz"
Jul 30 17:03:00.651634 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:03:00.651616    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c494027f-c3f1-4e9f-ba91-7f2c30580969-trusted-ca-bundle\") pod \"console-59855b657f-8qznz\" (UID: \"c494027f-c3f1-4e9f-ba91-7f2c30580969\") " pod="openshift-console/console-59855b657f-8qznz"
Jul 30 17:03:00.651687 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:03:00.651669    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c494027f-c3f1-4e9f-ba91-7f2c30580969-console-oauth-config\") pod \"console-59855b657f-8qznz\" (UID: \"c494027f-c3f1-4e9f-ba91-7f2c30580969\") " pod="openshift-console/console-59855b657f-8qznz"
Jul 30 17:03:00.651723 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:03:00.651704    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c494027f-c3f1-4e9f-ba91-7f2c30580969-oauth-serving-cert\") pod \"console-59855b657f-8qznz\" (UID: \"c494027f-c3f1-4e9f-ba91-7f2c30580969\") " pod="openshift-console/console-59855b657f-8qznz"
Jul 30 17:03:00.696638 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:03:00.696604    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:03:00.752821 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:03:00.752783    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c494027f-c3f1-4e9f-ba91-7f2c30580969-console-config\") pod \"console-59855b657f-8qznz\" (UID: \"c494027f-c3f1-4e9f-ba91-7f2c30580969\") " pod="openshift-console/console-59855b657f-8qznz"
Jul 30 17:03:00.752997 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:03:00.752826    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vczc\" (UniqueName: \"kubernetes.io/projected/c494027f-c3f1-4e9f-ba91-7f2c30580969-kube-api-access-8vczc\") pod \"console-59855b657f-8qznz\" (UID: \"c494027f-c3f1-4e9f-ba91-7f2c30580969\") " pod="openshift-console/console-59855b657f-8qznz"
Jul 30 17:03:00.752997 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:03:00.752858    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c494027f-c3f1-4e9f-ba91-7f2c30580969-service-ca\") pod \"console-59855b657f-8qznz\" (UID: \"c494027f-c3f1-4e9f-ba91-7f2c30580969\") " pod="openshift-console/console-59855b657f-8qznz"
Jul 30 17:03:00.752997 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:03:00.752892    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c494027f-c3f1-4e9f-ba91-7f2c30580969-console-serving-cert\") pod \"console-59855b657f-8qznz\" (UID: \"c494027f-c3f1-4e9f-ba91-7f2c30580969\") " pod="openshift-console/console-59855b657f-8qznz"
Jul 30 17:03:00.752997 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:03:00.752913    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c494027f-c3f1-4e9f-ba91-7f2c30580969-trusted-ca-bundle\") pod \"console-59855b657f-8qznz\" (UID: \"c494027f-c3f1-4e9f-ba91-7f2c30580969\") " pod="openshift-console/console-59855b657f-8qznz"
Jul 30 17:03:00.752997 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:03:00.752949    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c494027f-c3f1-4e9f-ba91-7f2c30580969-console-oauth-config\") pod \"console-59855b657f-8qznz\" (UID: \"c494027f-c3f1-4e9f-ba91-7f2c30580969\") " pod="openshift-console/console-59855b657f-8qznz"
Jul 30 17:03:00.752997 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:03:00.752979    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c494027f-c3f1-4e9f-ba91-7f2c30580969-oauth-serving-cert\") pod \"console-59855b657f-8qznz\" (UID: \"c494027f-c3f1-4e9f-ba91-7f2c30580969\") " pod="openshift-console/console-59855b657f-8qznz"
Jul 30 17:03:00.753642 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:03:00.753618    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c494027f-c3f1-4e9f-ba91-7f2c30580969-service-ca\") pod \"console-59855b657f-8qznz\" (UID: \"c494027f-c3f1-4e9f-ba91-7f2c30580969\") " pod="openshift-console/console-59855b657f-8qznz"
Jul 30 17:03:00.753889 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:03:00.753852    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c494027f-c3f1-4e9f-ba91-7f2c30580969-oauth-serving-cert\") pod \"console-59855b657f-8qznz\" (UID: \"c494027f-c3f1-4e9f-ba91-7f2c30580969\") " pod="openshift-console/console-59855b657f-8qznz"
Jul 30 17:03:00.753889 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:03:00.753884    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c494027f-c3f1-4e9f-ba91-7f2c30580969-console-config\") pod \"console-59855b657f-8qznz\" (UID: \"c494027f-c3f1-4e9f-ba91-7f2c30580969\") " pod="openshift-console/console-59855b657f-8qznz"
Jul 30 17:03:00.754100 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:03:00.753938    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c494027f-c3f1-4e9f-ba91-7f2c30580969-trusted-ca-bundle\") pod \"console-59855b657f-8qznz\" (UID: \"c494027f-c3f1-4e9f-ba91-7f2c30580969\") " pod="openshift-console/console-59855b657f-8qznz"
Jul 30 17:03:00.755046 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:03:00.755022    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:03:00.755578 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:03:00.755555    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c494027f-c3f1-4e9f-ba91-7f2c30580969-console-serving-cert\") pod \"console-59855b657f-8qznz\" (UID: \"c494027f-c3f1-4e9f-ba91-7f2c30580969\") " pod="openshift-console/console-59855b657f-8qznz"
Jul 30 17:03:00.755657 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:03:00.755642    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c494027f-c3f1-4e9f-ba91-7f2c30580969-console-oauth-config\") pod \"console-59855b657f-8qznz\" (UID: \"c494027f-c3f1-4e9f-ba91-7f2c30580969\") " pod="openshift-console/console-59855b657f-8qznz"
Jul 30 17:03:00.760860 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:03:00.760836    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vczc\" (UniqueName: \"kubernetes.io/projected/c494027f-c3f1-4e9f-ba91-7f2c30580969-kube-api-access-8vczc\") pod \"console-59855b657f-8qznz\" (UID: \"c494027f-c3f1-4e9f-ba91-7f2c30580969\") " pod="openshift-console/console-59855b657f-8qznz"
Jul 30 17:03:00.948237 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:03:00.948142    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59855b657f-8qznz"
Jul 30 17:03:01.078298 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:03:01.078277    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59855b657f-8qznz"]
Jul 30 17:03:01.080139 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:03:01.080114    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc494027f_c3f1_4e9f_ba91_7f2c30580969.slice/crio-6e2a23b97c74570b6acd91d856e0897e2b710ffa36ec9207219187e5e97673fe WatchSource:0}: Error finding container 6e2a23b97c74570b6acd91d856e0897e2b710ffa36ec9207219187e5e97673fe: Status 404 returned error can't find the container with id 6e2a23b97c74570b6acd91d856e0897e2b710ffa36ec9207219187e5e97673fe
Jul 30 17:03:01.825536 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:03:01.825504    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59855b657f-8qznz" event={"ID":"c494027f-c3f1-4e9f-ba91-7f2c30580969","Type":"ContainerStarted","Data":"16ce115fda1558cfbdbd46ea84d739f8384b3f32fb51e1e19219b2f23db298d2"}
Jul 30 17:03:01.825680 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:03:01.825552    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59855b657f-8qznz" event={"ID":"c494027f-c3f1-4e9f-ba91-7f2c30580969","Type":"ContainerStarted","Data":"6e2a23b97c74570b6acd91d856e0897e2b710ffa36ec9207219187e5e97673fe"}
Jul 30 17:03:01.843801 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:03:01.843737    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-59855b657f-8qznz" podStartSLOduration=1.843721113 podStartE2EDuration="1.843721113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:03:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:03:01.842933046 +0000 UTC m=+156.330300447" watchObservedRunningTime="2026-07-30 17:03:01.843721113 +0000 UTC m=+156.331088514"
Jul 30 17:03:10.948724 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:03:10.948678    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-59855b657f-8qznz"
Jul 30 17:03:10.949186 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:03:10.948765    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-59855b657f-8qznz"
Jul 30 17:03:10.953501 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:03:10.953480    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-59855b657f-8qznz"
Jul 30 17:03:11.874323 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:03:11.874299    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-59855b657f-8qznz"
Jul 30 17:03:11.924360 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:03:11.924329    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8447bbff85-pspxs"]
Jul 30 17:03:12.040716 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:03:12.040689    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:03:12.100820 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:03:12.100787    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:03:12.161810 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:03:12.161734    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:03:32.506030 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:03:32.505977    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:03:34.197655 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:03:34.197621    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:03:34.250165 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:03:34.250130    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:03:36.944436 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:03:36.944364    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-8447bbff85-pspxs" podUID="137b1ef1-19fb-49d3-878b-b6b85f333998" containerName="console" containerID="cri-o://fc27a012936ab47d9bebc2b92aaf89d8a9a5f4683b0297474ef427f75516a037" gracePeriod=15
Jul 30 17:03:37.186730 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:03:37.186708    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8447bbff85-pspxs"
Jul 30 17:03:37.280161 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:03:37.280068    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/137b1ef1-19fb-49d3-878b-b6b85f333998-service-ca\") pod \"137b1ef1-19fb-49d3-878b-b6b85f333998\" (UID: \"137b1ef1-19fb-49d3-878b-b6b85f333998\") "
Jul 30 17:03:37.280161 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:03:37.280119    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/137b1ef1-19fb-49d3-878b-b6b85f333998-console-serving-cert\") pod \"137b1ef1-19fb-49d3-878b-b6b85f333998\" (UID: \"137b1ef1-19fb-49d3-878b-b6b85f333998\") "
Jul 30 17:03:37.280161 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:03:37.280159    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/137b1ef1-19fb-49d3-878b-b6b85f333998-oauth-serving-cert\") pod \"137b1ef1-19fb-49d3-878b-b6b85f333998\" (UID: \"137b1ef1-19fb-49d3-878b-b6b85f333998\") "
Jul 30 17:03:37.280413 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:03:37.280269    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/137b1ef1-19fb-49d3-878b-b6b85f333998-console-oauth-config\") pod \"137b1ef1-19fb-49d3-878b-b6b85f333998\" (UID: \"137b1ef1-19fb-49d3-878b-b6b85f333998\") "
Jul 30 17:03:37.280413 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:03:37.280333    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g8n9w\" (UniqueName: \"kubernetes.io/projected/137b1ef1-19fb-49d3-878b-b6b85f333998-kube-api-access-g8n9w\") pod \"137b1ef1-19fb-49d3-878b-b6b85f333998\" (UID: \"137b1ef1-19fb-49d3-878b-b6b85f333998\") "
Jul 30 17:03:37.280413 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:03:37.280403    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/137b1ef1-19fb-49d3-878b-b6b85f333998-trusted-ca-bundle\") pod \"137b1ef1-19fb-49d3-878b-b6b85f333998\" (UID: \"137b1ef1-19fb-49d3-878b-b6b85f333998\") "
Jul 30 17:03:37.280567 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:03:37.280427    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/137b1ef1-19fb-49d3-878b-b6b85f333998-service-ca" (OuterVolumeSpecName: "service-ca") pod "137b1ef1-19fb-49d3-878b-b6b85f333998" (UID: "137b1ef1-19fb-49d3-878b-b6b85f333998"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:03:37.280567 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:03:37.280436    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/137b1ef1-19fb-49d3-878b-b6b85f333998-console-config\") pod \"137b1ef1-19fb-49d3-878b-b6b85f333998\" (UID: \"137b1ef1-19fb-49d3-878b-b6b85f333998\") "
Jul 30 17:03:37.280567 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:03:37.280536    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/137b1ef1-19fb-49d3-878b-b6b85f333998-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "137b1ef1-19fb-49d3-878b-b6b85f333998" (UID: "137b1ef1-19fb-49d3-878b-b6b85f333998"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:03:37.280762 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:03:37.280743    2555 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/137b1ef1-19fb-49d3-878b-b6b85f333998-service-ca\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:03:37.280810 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:03:37.280769    2555 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/137b1ef1-19fb-49d3-878b-b6b85f333998-oauth-serving-cert\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:03:37.280810 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:03:37.280784    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/137b1ef1-19fb-49d3-878b-b6b85f333998-console-config" (OuterVolumeSpecName: "console-config") pod "137b1ef1-19fb-49d3-878b-b6b85f333998" (UID: "137b1ef1-19fb-49d3-878b-b6b85f333998"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:03:37.280886 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:03:37.280801    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/137b1ef1-19fb-49d3-878b-b6b85f333998-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "137b1ef1-19fb-49d3-878b-b6b85f333998" (UID: "137b1ef1-19fb-49d3-878b-b6b85f333998"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:03:37.282515 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:03:37.282492    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/137b1ef1-19fb-49d3-878b-b6b85f333998-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "137b1ef1-19fb-49d3-878b-b6b85f333998" (UID: "137b1ef1-19fb-49d3-878b-b6b85f333998"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:03:37.282945 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:03:37.282913    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/137b1ef1-19fb-49d3-878b-b6b85f333998-kube-api-access-g8n9w" (OuterVolumeSpecName: "kube-api-access-g8n9w") pod "137b1ef1-19fb-49d3-878b-b6b85f333998" (UID: "137b1ef1-19fb-49d3-878b-b6b85f333998"). InnerVolumeSpecName "kube-api-access-g8n9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 17:03:37.282945 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:03:37.282929    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/137b1ef1-19fb-49d3-878b-b6b85f333998-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "137b1ef1-19fb-49d3-878b-b6b85f333998" (UID: "137b1ef1-19fb-49d3-878b-b6b85f333998"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:03:37.381370 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:03:37.381334    2555 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/137b1ef1-19fb-49d3-878b-b6b85f333998-trusted-ca-bundle\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:03:37.381370 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:03:37.381365    2555 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/137b1ef1-19fb-49d3-878b-b6b85f333998-console-config\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:03:37.381370 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:03:37.381374    2555 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/137b1ef1-19fb-49d3-878b-b6b85f333998-console-serving-cert\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:03:37.381588 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:03:37.381382    2555 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/137b1ef1-19fb-49d3-878b-b6b85f333998-console-oauth-config\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:03:37.381588 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:03:37.381391    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8n9w\" (UniqueName: \"kubernetes.io/projected/137b1ef1-19fb-49d3-878b-b6b85f333998-kube-api-access-g8n9w\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:03:37.969242 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:03:37.969209    2555 generic.go:356] "Generic (PLEG): container finished" podID="137b1ef1-19fb-49d3-878b-b6b85f333998" containerID="fc27a012936ab47d9bebc2b92aaf89d8a9a5f4683b0297474ef427f75516a037" exitCode=0
Jul 30 17:03:37.969242 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:03:37.969246    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8447bbff85-pspxs"
Jul 30 17:03:37.969697 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:03:37.969295    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8447bbff85-pspxs" event={"ID":"137b1ef1-19fb-49d3-878b-b6b85f333998","Type":"ContainerDied","Data":"fc27a012936ab47d9bebc2b92aaf89d8a9a5f4683b0297474ef427f75516a037"}
Jul 30 17:03:37.969697 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:03:37.969376    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8447bbff85-pspxs" event={"ID":"137b1ef1-19fb-49d3-878b-b6b85f333998","Type":"ContainerDied","Data":"40b463c54a68b545371f59ceb5526057adf08fc67b251db9f17d25ea6c52298d"}
Jul 30 17:03:37.969697 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:03:37.969397    2555 scope.go:117] "RemoveContainer" containerID="fc27a012936ab47d9bebc2b92aaf89d8a9a5f4683b0297474ef427f75516a037"
Jul 30 17:03:37.978080 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:03:37.978059    2555 scope.go:117] "RemoveContainer" containerID="fc27a012936ab47d9bebc2b92aaf89d8a9a5f4683b0297474ef427f75516a037"
Jul 30 17:03:37.978374 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:03:37.978353    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc27a012936ab47d9bebc2b92aaf89d8a9a5f4683b0297474ef427f75516a037\": container with ID starting with fc27a012936ab47d9bebc2b92aaf89d8a9a5f4683b0297474ef427f75516a037 not found: ID does not exist" containerID="fc27a012936ab47d9bebc2b92aaf89d8a9a5f4683b0297474ef427f75516a037"
Jul 30 17:03:37.978420 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:03:37.978385    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc27a012936ab47d9bebc2b92aaf89d8a9a5f4683b0297474ef427f75516a037"} err="failed to get container status \"fc27a012936ab47d9bebc2b92aaf89d8a9a5f4683b0297474ef427f75516a037\": rpc error: code = NotFound desc = could not find container \"fc27a012936ab47d9bebc2b92aaf89d8a9a5f4683b0297474ef427f75516a037\": container with ID starting with fc27a012936ab47d9bebc2b92aaf89d8a9a5f4683b0297474ef427f75516a037 not found: ID does not exist"
Jul 30 17:03:37.991991 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:03:37.991963    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8447bbff85-pspxs"]
Jul 30 17:03:37.993744 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:03:37.993721    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8447bbff85-pspxs"]
Jul 30 17:03:38.059920 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:03:38.059879    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="137b1ef1-19fb-49d3-878b-b6b85f333998" path="/var/lib/kubelet/pods/137b1ef1-19fb-49d3-878b-b6b85f333998/volumes"
Jul 30 17:04:06.309179 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:06.309145    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:04:06.366490 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:06.366451    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:04:10.435627 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:10.435594    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-68f68885b6-n7n7m"]
Jul 30 17:04:10.435993 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:10.435894    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="137b1ef1-19fb-49d3-878b-b6b85f333998" containerName="console"
Jul 30 17:04:10.435993 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:10.435905    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="137b1ef1-19fb-49d3-878b-b6b85f333998" containerName="console"
Jul 30 17:04:10.435993 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:10.435966    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="137b1ef1-19fb-49d3-878b-b6b85f333998" containerName="console"
Jul 30 17:04:10.438760 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:10.438745    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68f68885b6-n7n7m"
Jul 30 17:04:10.450694 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:10.450670    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68f68885b6-n7n7m"]
Jul 30 17:04:10.561605 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:10.561564    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3f441ce7-8a25-41b8-a8c1-28fb65d36f0f-console-config\") pod \"console-68f68885b6-n7n7m\" (UID: \"3f441ce7-8a25-41b8-a8c1-28fb65d36f0f\") " pod="openshift-console/console-68f68885b6-n7n7m"
Jul 30 17:04:10.561783 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:10.561625    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3f441ce7-8a25-41b8-a8c1-28fb65d36f0f-console-serving-cert\") pod \"console-68f68885b6-n7n7m\" (UID: \"3f441ce7-8a25-41b8-a8c1-28fb65d36f0f\") " pod="openshift-console/console-68f68885b6-n7n7m"
Jul 30 17:04:10.561783 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:10.561695    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3f441ce7-8a25-41b8-a8c1-28fb65d36f0f-console-oauth-config\") pod \"console-68f68885b6-n7n7m\" (UID: \"3f441ce7-8a25-41b8-a8c1-28fb65d36f0f\") " pod="openshift-console/console-68f68885b6-n7n7m"
Jul 30 17:04:10.561783 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:10.561728    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cgfh\" (UniqueName: \"kubernetes.io/projected/3f441ce7-8a25-41b8-a8c1-28fb65d36f0f-kube-api-access-9cgfh\") pod \"console-68f68885b6-n7n7m\" (UID: \"3f441ce7-8a25-41b8-a8c1-28fb65d36f0f\") " pod="openshift-console/console-68f68885b6-n7n7m"
Jul 30 17:04:10.561783 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:10.561748    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3f441ce7-8a25-41b8-a8c1-28fb65d36f0f-service-ca\") pod \"console-68f68885b6-n7n7m\" (UID: \"3f441ce7-8a25-41b8-a8c1-28fb65d36f0f\") " pod="openshift-console/console-68f68885b6-n7n7m"
Jul 30 17:04:10.561783 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:10.561775    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3f441ce7-8a25-41b8-a8c1-28fb65d36f0f-oauth-serving-cert\") pod \"console-68f68885b6-n7n7m\" (UID: \"3f441ce7-8a25-41b8-a8c1-28fb65d36f0f\") " pod="openshift-console/console-68f68885b6-n7n7m"
Jul 30 17:04:10.561980 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:10.561855    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f441ce7-8a25-41b8-a8c1-28fb65d36f0f-trusted-ca-bundle\") pod \"console-68f68885b6-n7n7m\" (UID: \"3f441ce7-8a25-41b8-a8c1-28fb65d36f0f\") " pod="openshift-console/console-68f68885b6-n7n7m"
Jul 30 17:04:10.662453 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:10.662410    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3f441ce7-8a25-41b8-a8c1-28fb65d36f0f-oauth-serving-cert\") pod \"console-68f68885b6-n7n7m\" (UID: \"3f441ce7-8a25-41b8-a8c1-28fb65d36f0f\") " pod="openshift-console/console-68f68885b6-n7n7m"
Jul 30 17:04:10.662611 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:10.662463    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f441ce7-8a25-41b8-a8c1-28fb65d36f0f-trusted-ca-bundle\") pod \"console-68f68885b6-n7n7m\" (UID: \"3f441ce7-8a25-41b8-a8c1-28fb65d36f0f\") " pod="openshift-console/console-68f68885b6-n7n7m"
Jul 30 17:04:10.662611 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:10.662497    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3f441ce7-8a25-41b8-a8c1-28fb65d36f0f-console-config\") pod \"console-68f68885b6-n7n7m\" (UID: \"3f441ce7-8a25-41b8-a8c1-28fb65d36f0f\") " pod="openshift-console/console-68f68885b6-n7n7m"
Jul 30 17:04:10.662611 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:10.662530    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3f441ce7-8a25-41b8-a8c1-28fb65d36f0f-console-serving-cert\") pod \"console-68f68885b6-n7n7m\" (UID: \"3f441ce7-8a25-41b8-a8c1-28fb65d36f0f\") " pod="openshift-console/console-68f68885b6-n7n7m"
Jul 30 17:04:10.662611 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:10.662559    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3f441ce7-8a25-41b8-a8c1-28fb65d36f0f-console-oauth-config\") pod \"console-68f68885b6-n7n7m\" (UID: \"3f441ce7-8a25-41b8-a8c1-28fb65d36f0f\") " pod="openshift-console/console-68f68885b6-n7n7m"
Jul 30 17:04:10.662611 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:10.662586    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cgfh\" (UniqueName: \"kubernetes.io/projected/3f441ce7-8a25-41b8-a8c1-28fb65d36f0f-kube-api-access-9cgfh\") pod \"console-68f68885b6-n7n7m\" (UID: \"3f441ce7-8a25-41b8-a8c1-28fb65d36f0f\") " pod="openshift-console/console-68f68885b6-n7n7m"
Jul 30 17:04:10.662844 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:10.662611    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3f441ce7-8a25-41b8-a8c1-28fb65d36f0f-service-ca\") pod \"console-68f68885b6-n7n7m\" (UID: \"3f441ce7-8a25-41b8-a8c1-28fb65d36f0f\") " pod="openshift-console/console-68f68885b6-n7n7m"
Jul 30 17:04:10.663259 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:10.663233    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3f441ce7-8a25-41b8-a8c1-28fb65d36f0f-oauth-serving-cert\") pod \"console-68f68885b6-n7n7m\" (UID: \"3f441ce7-8a25-41b8-a8c1-28fb65d36f0f\") " pod="openshift-console/console-68f68885b6-n7n7m"
Jul 30 17:04:10.663377 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:10.663292    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3f441ce7-8a25-41b8-a8c1-28fb65d36f0f-console-config\") pod \"console-68f68885b6-n7n7m\" (UID: \"3f441ce7-8a25-41b8-a8c1-28fb65d36f0f\") " pod="openshift-console/console-68f68885b6-n7n7m"
Jul 30 17:04:10.663377 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:10.663356    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3f441ce7-8a25-41b8-a8c1-28fb65d36f0f-service-ca\") pod \"console-68f68885b6-n7n7m\" (UID: \"3f441ce7-8a25-41b8-a8c1-28fb65d36f0f\") " pod="openshift-console/console-68f68885b6-n7n7m"
Jul 30 17:04:10.663481 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:10.663440    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f441ce7-8a25-41b8-a8c1-28fb65d36f0f-trusted-ca-bundle\") pod \"console-68f68885b6-n7n7m\" (UID: \"3f441ce7-8a25-41b8-a8c1-28fb65d36f0f\") " pod="openshift-console/console-68f68885b6-n7n7m"
Jul 30 17:04:10.665097 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:10.665070    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3f441ce7-8a25-41b8-a8c1-28fb65d36f0f-console-serving-cert\") pod \"console-68f68885b6-n7n7m\" (UID: \"3f441ce7-8a25-41b8-a8c1-28fb65d36f0f\") " pod="openshift-console/console-68f68885b6-n7n7m"
Jul 30 17:04:10.665211 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:10.665168    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3f441ce7-8a25-41b8-a8c1-28fb65d36f0f-console-oauth-config\") pod \"console-68f68885b6-n7n7m\" (UID: \"3f441ce7-8a25-41b8-a8c1-28fb65d36f0f\") " pod="openshift-console/console-68f68885b6-n7n7m"
Jul 30 17:04:10.671614 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:10.671596    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cgfh\" (UniqueName: \"kubernetes.io/projected/3f441ce7-8a25-41b8-a8c1-28fb65d36f0f-kube-api-access-9cgfh\") pod \"console-68f68885b6-n7n7m\" (UID: \"3f441ce7-8a25-41b8-a8c1-28fb65d36f0f\") " pod="openshift-console/console-68f68885b6-n7n7m"
Jul 30 17:04:10.747184 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:10.747094    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68f68885b6-n7n7m"
Jul 30 17:04:10.876200 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:10.876177    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68f68885b6-n7n7m"]
Jul 30 17:04:10.879630 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:04:10.879599    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f441ce7_8a25_41b8_a8c1_28fb65d36f0f.slice/crio-cee505941ce06740922c5e679923433d6f299aeed2bb8597f8b3dd96402a68dc WatchSource:0}: Error finding container cee505941ce06740922c5e679923433d6f299aeed2bb8597f8b3dd96402a68dc: Status 404 returned error can't find the container with id cee505941ce06740922c5e679923433d6f299aeed2bb8597f8b3dd96402a68dc
Jul 30 17:04:11.091035 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:11.090971    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68f68885b6-n7n7m" event={"ID":"3f441ce7-8a25-41b8-a8c1-28fb65d36f0f","Type":"ContainerStarted","Data":"75a7d95e94f57391b46552fcea8786da569fff1ecb5e8039d0f1b52f9c61ee29"}
Jul 30 17:04:11.091035 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:11.091041    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68f68885b6-n7n7m" event={"ID":"3f441ce7-8a25-41b8-a8c1-28fb65d36f0f","Type":"ContainerStarted","Data":"cee505941ce06740922c5e679923433d6f299aeed2bb8597f8b3dd96402a68dc"}
Jul 30 17:04:11.109888 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:11.109834    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-68f68885b6-n7n7m" podStartSLOduration=1.109817245 podStartE2EDuration="1.109817245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:04:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:04:11.109243602 +0000 UTC m=+225.596611003" watchObservedRunningTime="2026-07-30 17:04:11.109817245 +0000 UTC m=+225.597184645"
Jul 30 17:04:11.177870 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:11.177842    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:04:11.228748 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:11.228717    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:04:18.289650 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:18.289621    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:04:18.359901 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:18.359865    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:04:20.747369 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:20.747322    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-68f68885b6-n7n7m"
Jul 30 17:04:20.747787 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:20.747392    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-68f68885b6-n7n7m"
Jul 30 17:04:20.752083 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:20.752064    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-68f68885b6-n7n7m"
Jul 30 17:04:21.132986 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:21.132961    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-68f68885b6-n7n7m"
Jul 30 17:04:21.185215 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:21.185186    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59855b657f-8qznz"]
Jul 30 17:04:21.309245 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:21.309216    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:04:21.409859 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:21.409782    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:04:21.465813 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:21.465783    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:04:24.895681 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:24.895648    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:04:24.995836 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:24.995800    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:04:43.981966 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:43.981934    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:04:46.207683 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:46.207648    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-59855b657f-8qznz" podUID="c494027f-c3f1-4e9f-ba91-7f2c30580969" containerName="console" containerID="cri-o://16ce115fda1558cfbdbd46ea84d739f8384b3f32fb51e1e19219b2f23db298d2" gracePeriod=15
Jul 30 17:04:46.450275 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:46.450253    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59855b657f-8qznz"
Jul 30 17:04:46.547400 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:46.547376    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c494027f-c3f1-4e9f-ba91-7f2c30580969-service-ca\") pod \"c494027f-c3f1-4e9f-ba91-7f2c30580969\" (UID: \"c494027f-c3f1-4e9f-ba91-7f2c30580969\") "
Jul 30 17:04:46.547570 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:46.547417    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c494027f-c3f1-4e9f-ba91-7f2c30580969-console-serving-cert\") pod \"c494027f-c3f1-4e9f-ba91-7f2c30580969\" (UID: \"c494027f-c3f1-4e9f-ba91-7f2c30580969\") "
Jul 30 17:04:46.547570 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:46.547459    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c494027f-c3f1-4e9f-ba91-7f2c30580969-console-oauth-config\") pod \"c494027f-c3f1-4e9f-ba91-7f2c30580969\" (UID: \"c494027f-c3f1-4e9f-ba91-7f2c30580969\") "
Jul 30 17:04:46.547570 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:46.547498    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8vczc\" (UniqueName: \"kubernetes.io/projected/c494027f-c3f1-4e9f-ba91-7f2c30580969-kube-api-access-8vczc\") pod \"c494027f-c3f1-4e9f-ba91-7f2c30580969\" (UID: \"c494027f-c3f1-4e9f-ba91-7f2c30580969\") "
Jul 30 17:04:46.547570 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:46.547527    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c494027f-c3f1-4e9f-ba91-7f2c30580969-trusted-ca-bundle\") pod \"c494027f-c3f1-4e9f-ba91-7f2c30580969\" (UID: \"c494027f-c3f1-4e9f-ba91-7f2c30580969\") "
Jul 30 17:04:46.547570 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:46.547558    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c494027f-c3f1-4e9f-ba91-7f2c30580969-oauth-serving-cert\") pod \"c494027f-c3f1-4e9f-ba91-7f2c30580969\" (UID: \"c494027f-c3f1-4e9f-ba91-7f2c30580969\") "
Jul 30 17:04:46.547819 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:46.547588    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c494027f-c3f1-4e9f-ba91-7f2c30580969-console-config\") pod \"c494027f-c3f1-4e9f-ba91-7f2c30580969\" (UID: \"c494027f-c3f1-4e9f-ba91-7f2c30580969\") "
Jul 30 17:04:46.547969 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:46.547919    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c494027f-c3f1-4e9f-ba91-7f2c30580969-service-ca" (OuterVolumeSpecName: "service-ca") pod "c494027f-c3f1-4e9f-ba91-7f2c30580969" (UID: "c494027f-c3f1-4e9f-ba91-7f2c30580969"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:04:46.548119 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:46.548071    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c494027f-c3f1-4e9f-ba91-7f2c30580969-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c494027f-c3f1-4e9f-ba91-7f2c30580969" (UID: "c494027f-c3f1-4e9f-ba91-7f2c30580969"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:04:46.548119 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:46.548088    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c494027f-c3f1-4e9f-ba91-7f2c30580969-console-config" (OuterVolumeSpecName: "console-config") pod "c494027f-c3f1-4e9f-ba91-7f2c30580969" (UID: "c494027f-c3f1-4e9f-ba91-7f2c30580969"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:04:46.548119 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:46.548079    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c494027f-c3f1-4e9f-ba91-7f2c30580969-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "c494027f-c3f1-4e9f-ba91-7f2c30580969" (UID: "c494027f-c3f1-4e9f-ba91-7f2c30580969"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:04:46.549770 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:46.549749    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c494027f-c3f1-4e9f-ba91-7f2c30580969-kube-api-access-8vczc" (OuterVolumeSpecName: "kube-api-access-8vczc") pod "c494027f-c3f1-4e9f-ba91-7f2c30580969" (UID: "c494027f-c3f1-4e9f-ba91-7f2c30580969"). InnerVolumeSpecName "kube-api-access-8vczc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 17:04:46.550136 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:46.550110    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c494027f-c3f1-4e9f-ba91-7f2c30580969-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c494027f-c3f1-4e9f-ba91-7f2c30580969" (UID: "c494027f-c3f1-4e9f-ba91-7f2c30580969"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:04:46.550217 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:46.550131    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c494027f-c3f1-4e9f-ba91-7f2c30580969-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c494027f-c3f1-4e9f-ba91-7f2c30580969" (UID: "c494027f-c3f1-4e9f-ba91-7f2c30580969"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:04:46.648758 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:46.648708    2555 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c494027f-c3f1-4e9f-ba91-7f2c30580969-service-ca\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:04:46.648758 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:46.648751    2555 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c494027f-c3f1-4e9f-ba91-7f2c30580969-console-serving-cert\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:04:46.648758 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:46.648762    2555 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c494027f-c3f1-4e9f-ba91-7f2c30580969-console-oauth-config\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:04:46.648758 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:46.648771    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vczc\" (UniqueName: \"kubernetes.io/projected/c494027f-c3f1-4e9f-ba91-7f2c30580969-kube-api-access-8vczc\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:04:46.649064 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:46.648782    2555 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c494027f-c3f1-4e9f-ba91-7f2c30580969-trusted-ca-bundle\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:04:46.649064 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:46.648790    2555 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c494027f-c3f1-4e9f-ba91-7f2c30580969-oauth-serving-cert\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:04:46.649064 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:46.648798    2555 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c494027f-c3f1-4e9f-ba91-7f2c30580969-console-config\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:04:47.232669 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:47.232628    2555 generic.go:356] "Generic (PLEG): container finished" podID="c494027f-c3f1-4e9f-ba91-7f2c30580969" containerID="16ce115fda1558cfbdbd46ea84d739f8384b3f32fb51e1e19219b2f23db298d2" exitCode=0
Jul 30 17:04:47.233086 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:47.232678    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59855b657f-8qznz"
Jul 30 17:04:47.233086 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:47.232688    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59855b657f-8qznz" event={"ID":"c494027f-c3f1-4e9f-ba91-7f2c30580969","Type":"ContainerDied","Data":"16ce115fda1558cfbdbd46ea84d739f8384b3f32fb51e1e19219b2f23db298d2"}
Jul 30 17:04:47.233086 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:47.232736    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59855b657f-8qznz" event={"ID":"c494027f-c3f1-4e9f-ba91-7f2c30580969","Type":"ContainerDied","Data":"6e2a23b97c74570b6acd91d856e0897e2b710ffa36ec9207219187e5e97673fe"}
Jul 30 17:04:47.233086 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:47.232763    2555 scope.go:117] "RemoveContainer" containerID="16ce115fda1558cfbdbd46ea84d739f8384b3f32fb51e1e19219b2f23db298d2"
Jul 30 17:04:47.241185 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:47.241164    2555 scope.go:117] "RemoveContainer" containerID="16ce115fda1558cfbdbd46ea84d739f8384b3f32fb51e1e19219b2f23db298d2"
Jul 30 17:04:47.241426 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:04:47.241408    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16ce115fda1558cfbdbd46ea84d739f8384b3f32fb51e1e19219b2f23db298d2\": container with ID starting with 16ce115fda1558cfbdbd46ea84d739f8384b3f32fb51e1e19219b2f23db298d2 not found: ID does not exist" containerID="16ce115fda1558cfbdbd46ea84d739f8384b3f32fb51e1e19219b2f23db298d2"
Jul 30 17:04:47.241470 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:47.241435    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16ce115fda1558cfbdbd46ea84d739f8384b3f32fb51e1e19219b2f23db298d2"} err="failed to get container status \"16ce115fda1558cfbdbd46ea84d739f8384b3f32fb51e1e19219b2f23db298d2\": rpc error: code = NotFound desc = could not find container \"16ce115fda1558cfbdbd46ea84d739f8384b3f32fb51e1e19219b2f23db298d2\": container with ID starting with 16ce115fda1558cfbdbd46ea84d739f8384b3f32fb51e1e19219b2f23db298d2 not found: ID does not exist"
Jul 30 17:04:47.255236 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:47.255208    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59855b657f-8qznz"]
Jul 30 17:04:47.262312 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:47.262288    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-59855b657f-8qznz"]
Jul 30 17:04:48.059491 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:04:48.059460    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c494027f-c3f1-4e9f-ba91-7f2c30580969" path="/var/lib/kubelet/pods/c494027f-c3f1-4e9f-ba91-7f2c30580969/volumes"
Jul 30 17:05:11.176492 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:11.176461    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:05:11.351896 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:11.351866    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hx7nb"]
Jul 30 17:05:11.352198 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:11.352186    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c494027f-c3f1-4e9f-ba91-7f2c30580969" containerName="console"
Jul 30 17:05:11.352251 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:11.352199    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c494027f-c3f1-4e9f-ba91-7f2c30580969" containerName="console"
Jul 30 17:05:11.352305 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:11.352252    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c494027f-c3f1-4e9f-ba91-7f2c30580969" containerName="console"
Jul 30 17:05:11.356600 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:11.356581    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hx7nb"
Jul 30 17:05:11.359254 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:11.359233    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 30 17:05:11.360294 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:11.360273    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 30 17:05:11.360401 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:11.360312    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-85zn2\""
Jul 30 17:05:11.364549 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:11.364527    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hx7nb"]
Jul 30 17:05:11.462745 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:11.462650    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2rdz\" (UniqueName: \"kubernetes.io/projected/b5ae537c-1d5b-4e82-8f4a-203acb060d21-kube-api-access-s2rdz\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hx7nb\" (UID: \"b5ae537c-1d5b-4e82-8f4a-203acb060d21\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hx7nb"
Jul 30 17:05:11.462745 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:11.462691    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b5ae537c-1d5b-4e82-8f4a-203acb060d21-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hx7nb\" (UID: \"b5ae537c-1d5b-4e82-8f4a-203acb060d21\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hx7nb"
Jul 30 17:05:11.462961 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:11.462804    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b5ae537c-1d5b-4e82-8f4a-203acb060d21-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hx7nb\" (UID: \"b5ae537c-1d5b-4e82-8f4a-203acb060d21\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hx7nb"
Jul 30 17:05:11.563348 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:11.563310    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b5ae537c-1d5b-4e82-8f4a-203acb060d21-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hx7nb\" (UID: \"b5ae537c-1d5b-4e82-8f4a-203acb060d21\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hx7nb"
Jul 30 17:05:11.563514 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:11.563386    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2rdz\" (UniqueName: \"kubernetes.io/projected/b5ae537c-1d5b-4e82-8f4a-203acb060d21-kube-api-access-s2rdz\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hx7nb\" (UID: \"b5ae537c-1d5b-4e82-8f4a-203acb060d21\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hx7nb"
Jul 30 17:05:11.563514 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:11.563417    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b5ae537c-1d5b-4e82-8f4a-203acb060d21-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hx7nb\" (UID: \"b5ae537c-1d5b-4e82-8f4a-203acb060d21\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hx7nb"
Jul 30 17:05:11.563777 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:11.563757    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b5ae537c-1d5b-4e82-8f4a-203acb060d21-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hx7nb\" (UID: \"b5ae537c-1d5b-4e82-8f4a-203acb060d21\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hx7nb"
Jul 30 17:05:11.563777 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:11.563770    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b5ae537c-1d5b-4e82-8f4a-203acb060d21-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hx7nb\" (UID: \"b5ae537c-1d5b-4e82-8f4a-203acb060d21\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hx7nb"
Jul 30 17:05:11.573086 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:11.573065    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2rdz\" (UniqueName: \"kubernetes.io/projected/b5ae537c-1d5b-4e82-8f4a-203acb060d21-kube-api-access-s2rdz\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hx7nb\" (UID: \"b5ae537c-1d5b-4e82-8f4a-203acb060d21\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hx7nb"
Jul 30 17:05:11.666730 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:11.666696    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hx7nb"
Jul 30 17:05:11.791842 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:11.791634    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hx7nb"]
Jul 30 17:05:11.794019 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:05:11.793984    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5ae537c_1d5b_4e82_8f4a_203acb060d21.slice/crio-2eb8d60956a1cf3453af7ee14cd58760ecc865959f371d7da720379b020731c1 WatchSource:0}: Error finding container 2eb8d60956a1cf3453af7ee14cd58760ecc865959f371d7da720379b020731c1: Status 404 returned error can't find the container with id 2eb8d60956a1cf3453af7ee14cd58760ecc865959f371d7da720379b020731c1
Jul 30 17:05:12.330853 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:12.330814    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hx7nb" event={"ID":"b5ae537c-1d5b-4e82-8f4a-203acb060d21","Type":"ContainerStarted","Data":"2eb8d60956a1cf3453af7ee14cd58760ecc865959f371d7da720379b020731c1"}
Jul 30 17:05:12.643290 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:12.643209    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:05:18.356843 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:18.356805    2555 generic.go:356] "Generic (PLEG): container finished" podID="b5ae537c-1d5b-4e82-8f4a-203acb060d21" containerID="b4760104d442c3a11343a867841d3e8c0b9465585708b640e2eecb41654d4501" exitCode=0
Jul 30 17:05:18.357291 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:18.356857    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hx7nb" event={"ID":"b5ae537c-1d5b-4e82-8f4a-203acb060d21","Type":"ContainerDied","Data":"b4760104d442c3a11343a867841d3e8c0b9465585708b640e2eecb41654d4501"}
Jul 30 17:05:20.366392 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:20.366356    2555 generic.go:356] "Generic (PLEG): container finished" podID="b5ae537c-1d5b-4e82-8f4a-203acb060d21" containerID="e5e203b7fe3b159bb986277786d4f2e5158734057c4c1c7ccc0aecb001dbc2f7" exitCode=0
Jul 30 17:05:20.366392 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:20.366395    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hx7nb" event={"ID":"b5ae537c-1d5b-4e82-8f4a-203acb060d21","Type":"ContainerDied","Data":"e5e203b7fe3b159bb986277786d4f2e5158734057c4c1c7ccc0aecb001dbc2f7"}
Jul 30 17:05:27.404003 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:27.403973    2555 kubelet.go:1648] "Image garbage collection succeeded"
Jul 30 17:05:28.402296 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:28.402257    2555 generic.go:356] "Generic (PLEG): container finished" podID="b5ae537c-1d5b-4e82-8f4a-203acb060d21" containerID="7828dac9a1b73098aff94ca13e33671079f7d47277a49d31f524e7e41626ca18" exitCode=0
Jul 30 17:05:28.402465 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:28.402349    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hx7nb" event={"ID":"b5ae537c-1d5b-4e82-8f4a-203acb060d21","Type":"ContainerDied","Data":"7828dac9a1b73098aff94ca13e33671079f7d47277a49d31f524e7e41626ca18"}
Jul 30 17:05:29.526936 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:29.526913    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hx7nb"
Jul 30 17:05:29.624539 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:29.624506    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b5ae537c-1d5b-4e82-8f4a-203acb060d21-util\") pod \"b5ae537c-1d5b-4e82-8f4a-203acb060d21\" (UID: \"b5ae537c-1d5b-4e82-8f4a-203acb060d21\") "
Jul 30 17:05:29.624707 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:29.624607    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b5ae537c-1d5b-4e82-8f4a-203acb060d21-bundle\") pod \"b5ae537c-1d5b-4e82-8f4a-203acb060d21\" (UID: \"b5ae537c-1d5b-4e82-8f4a-203acb060d21\") "
Jul 30 17:05:29.624707 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:29.624634    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s2rdz\" (UniqueName: \"kubernetes.io/projected/b5ae537c-1d5b-4e82-8f4a-203acb060d21-kube-api-access-s2rdz\") pod \"b5ae537c-1d5b-4e82-8f4a-203acb060d21\" (UID: \"b5ae537c-1d5b-4e82-8f4a-203acb060d21\") "
Jul 30 17:05:29.625246 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:29.625212    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5ae537c-1d5b-4e82-8f4a-203acb060d21-bundle" (OuterVolumeSpecName: "bundle") pod "b5ae537c-1d5b-4e82-8f4a-203acb060d21" (UID: "b5ae537c-1d5b-4e82-8f4a-203acb060d21"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 17:05:29.626879 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:29.626853    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5ae537c-1d5b-4e82-8f4a-203acb060d21-kube-api-access-s2rdz" (OuterVolumeSpecName: "kube-api-access-s2rdz") pod "b5ae537c-1d5b-4e82-8f4a-203acb060d21" (UID: "b5ae537c-1d5b-4e82-8f4a-203acb060d21"). InnerVolumeSpecName "kube-api-access-s2rdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 17:05:29.630519 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:29.630469    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5ae537c-1d5b-4e82-8f4a-203acb060d21-util" (OuterVolumeSpecName: "util") pod "b5ae537c-1d5b-4e82-8f4a-203acb060d21" (UID: "b5ae537c-1d5b-4e82-8f4a-203acb060d21"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 17:05:29.726038 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:29.725926    2555 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b5ae537c-1d5b-4e82-8f4a-203acb060d21-util\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:05:29.726038 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:29.725967    2555 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b5ae537c-1d5b-4e82-8f4a-203acb060d21-bundle\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:05:29.726038 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:29.725977    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2rdz\" (UniqueName: \"kubernetes.io/projected/b5ae537c-1d5b-4e82-8f4a-203acb060d21-kube-api-access-s2rdz\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:05:30.412145 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:30.412074    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hx7nb"
Jul 30 17:05:30.412145 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:30.412081    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hx7nb" event={"ID":"b5ae537c-1d5b-4e82-8f4a-203acb060d21","Type":"ContainerDied","Data":"2eb8d60956a1cf3453af7ee14cd58760ecc865959f371d7da720379b020731c1"}
Jul 30 17:05:30.412145 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:30.412123    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2eb8d60956a1cf3453af7ee14cd58760ecc865959f371d7da720379b020731c1"
Jul 30 17:05:32.307301 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:32.307269    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:05:32.406406 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:32.406372    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:05:33.607813 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:33.607782    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-8wwx9"]
Jul 30 17:05:33.608326 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:33.608262    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5ae537c-1d5b-4e82-8f4a-203acb060d21" containerName="util"
Jul 30 17:05:33.608326 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:33.608279    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5ae537c-1d5b-4e82-8f4a-203acb060d21" containerName="util"
Jul 30 17:05:33.608326 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:33.608295    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5ae537c-1d5b-4e82-8f4a-203acb060d21" containerName="extract"
Jul 30 17:05:33.608326 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:33.608303    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5ae537c-1d5b-4e82-8f4a-203acb060d21" containerName="extract"
Jul 30 17:05:33.608531 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:33.608330    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5ae537c-1d5b-4e82-8f4a-203acb060d21" containerName="pull"
Jul 30 17:05:33.608531 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:33.608339    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5ae537c-1d5b-4e82-8f4a-203acb060d21" containerName="pull"
Jul 30 17:05:33.608531 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:33.608409    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5ae537c-1d5b-4e82-8f4a-203acb060d21" containerName="extract"
Jul 30 17:05:33.611384 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:33.611363    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-8wwx9"
Jul 30 17:05:33.614102 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:33.614079    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 30 17:05:33.614102 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:33.614093    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-k4t2k\""
Jul 30 17:05:33.614298 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:33.614082    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 30 17:05:33.614364 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:33.614344    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 30 17:05:33.622807 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:33.622784    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-8wwx9"]
Jul 30 17:05:33.654795 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:33.654752    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/ba6f27ee-ba26-46ec-b732-a063a5c7d8d1-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-8wwx9\" (UID: \"ba6f27ee-ba26-46ec-b732-a063a5c7d8d1\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-8wwx9"
Jul 30 17:05:33.654946 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:33.654818    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8d6k\" (UniqueName: \"kubernetes.io/projected/ba6f27ee-ba26-46ec-b732-a063a5c7d8d1-kube-api-access-q8d6k\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-8wwx9\" (UID: \"ba6f27ee-ba26-46ec-b732-a063a5c7d8d1\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-8wwx9"
Jul 30 17:05:33.756175 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:33.756135    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/ba6f27ee-ba26-46ec-b732-a063a5c7d8d1-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-8wwx9\" (UID: \"ba6f27ee-ba26-46ec-b732-a063a5c7d8d1\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-8wwx9"
Jul 30 17:05:33.756175 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:33.756183    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8d6k\" (UniqueName: \"kubernetes.io/projected/ba6f27ee-ba26-46ec-b732-a063a5c7d8d1-kube-api-access-q8d6k\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-8wwx9\" (UID: \"ba6f27ee-ba26-46ec-b732-a063a5c7d8d1\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-8wwx9"
Jul 30 17:05:33.758416 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:33.758394    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/ba6f27ee-ba26-46ec-b732-a063a5c7d8d1-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-8wwx9\" (UID: \"ba6f27ee-ba26-46ec-b732-a063a5c7d8d1\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-8wwx9"
Jul 30 17:05:33.765987 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:33.765966    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8d6k\" (UniqueName: \"kubernetes.io/projected/ba6f27ee-ba26-46ec-b732-a063a5c7d8d1-kube-api-access-q8d6k\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-8wwx9\" (UID: \"ba6f27ee-ba26-46ec-b732-a063a5c7d8d1\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-8wwx9"
Jul 30 17:05:33.921026 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:33.920923    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-8wwx9"
Jul 30 17:05:34.050900 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:34.050870    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-8wwx9"]
Jul 30 17:05:34.054224 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:05:34.054198    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba6f27ee_ba26_46ec_b732_a063a5c7d8d1.slice/crio-ffae6906b8271a9bf903d709796d56b7eff8a1ed8129ff1178feb69d29cf78f5 WatchSource:0}: Error finding container ffae6906b8271a9bf903d709796d56b7eff8a1ed8129ff1178feb69d29cf78f5: Status 404 returned error can't find the container with id ffae6906b8271a9bf903d709796d56b7eff8a1ed8129ff1178feb69d29cf78f5
Jul 30 17:05:34.055962 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:34.055945    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 30 17:05:34.428353 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:34.428316    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-8wwx9" event={"ID":"ba6f27ee-ba26-46ec-b732-a063a5c7d8d1","Type":"ContainerStarted","Data":"ffae6906b8271a9bf903d709796d56b7eff8a1ed8129ff1178feb69d29cf78f5"}
Jul 30 17:05:37.195849 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:37.195766    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:05:37.445152 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:37.445112    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-8wwx9" event={"ID":"ba6f27ee-ba26-46ec-b732-a063a5c7d8d1","Type":"ContainerStarted","Data":"122b06bae8c682cdc7991b05eb7d2647884ef7d86997a1807cf06fc2415b444a"}
Jul 30 17:05:37.445365 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:37.445286    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-8wwx9"
Jul 30 17:05:37.468799 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:37.468717    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-8wwx9" podStartSLOduration=1.7139569369999998 podStartE2EDuration="4.468702276s" totalImagesPullingTime="2.754745339s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:05:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:05:37.466745808 +0000 UTC m=+311.954113209" watchObservedRunningTime="2026-07-30 17:05:37.468702276 +0000 UTC m=+311.956069677"
Jul 30 17:05:37.478242 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:37.478217    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:05:37.484874 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:37.484854    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-77f648d694-7rshh"]
Jul 30 17:05:37.488161 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:37.488146    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-7rshh"
Jul 30 17:05:37.490470 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:37.490450    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 30 17:05:37.490581 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:37.490529    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Jul 30 17:05:37.490581 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:37.490566    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-srpx8\""
Jul 30 17:05:37.499483 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:37.499461    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-7rshh"]
Jul 30 17:05:37.587703 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:37.587574    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/fd8a1bec-7f8f-43ba-bce8-db1592ce379e-cabundle0\") pod \"keda-operator-77f648d694-7rshh\" (UID: \"fd8a1bec-7f8f-43ba-bce8-db1592ce379e\") " pod="openshift-keda/keda-operator-77f648d694-7rshh"
Jul 30 17:05:37.587921 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:37.587632    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn56t\" (UniqueName: \"kubernetes.io/projected/fd8a1bec-7f8f-43ba-bce8-db1592ce379e-kube-api-access-vn56t\") pod \"keda-operator-77f648d694-7rshh\" (UID: \"fd8a1bec-7f8f-43ba-bce8-db1592ce379e\") " pod="openshift-keda/keda-operator-77f648d694-7rshh"
Jul 30 17:05:37.588079 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:37.588051    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/fd8a1bec-7f8f-43ba-bce8-db1592ce379e-certificates\") pod \"keda-operator-77f648d694-7rshh\" (UID: \"fd8a1bec-7f8f-43ba-bce8-db1592ce379e\") " pod="openshift-keda/keda-operator-77f648d694-7rshh"
Jul 30 17:05:37.684695 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:37.684662    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:05:37.689265 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:37.689241    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/fd8a1bec-7f8f-43ba-bce8-db1592ce379e-cabundle0\") pod \"keda-operator-77f648d694-7rshh\" (UID: \"fd8a1bec-7f8f-43ba-bce8-db1592ce379e\") " pod="openshift-keda/keda-operator-77f648d694-7rshh"
Jul 30 17:05:37.689384 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:37.689272    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn56t\" (UniqueName: \"kubernetes.io/projected/fd8a1bec-7f8f-43ba-bce8-db1592ce379e-kube-api-access-vn56t\") pod \"keda-operator-77f648d694-7rshh\" (UID: \"fd8a1bec-7f8f-43ba-bce8-db1592ce379e\") " pod="openshift-keda/keda-operator-77f648d694-7rshh"
Jul 30 17:05:37.689384 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:37.689293    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/fd8a1bec-7f8f-43ba-bce8-db1592ce379e-certificates\") pod \"keda-operator-77f648d694-7rshh\" (UID: \"fd8a1bec-7f8f-43ba-bce8-db1592ce379e\") " pod="openshift-keda/keda-operator-77f648d694-7rshh"
Jul 30 17:05:37.689493 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:05:37.689475    2555 projected.go:266] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found
Jul 30 17:05:37.689528 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:05:37.689499    2555 secret.go:281] references non-existent secret key: ca.crt
Jul 30 17:05:37.689528 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:05:37.689507    2555 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 30 17:05:37.689528 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:05:37.689519    2555 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-77f648d694-7rshh: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Jul 30 17:05:37.689633 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:05:37.689569    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd8a1bec-7f8f-43ba-bce8-db1592ce379e-certificates podName:fd8a1bec-7f8f-43ba-bce8-db1592ce379e nodeName:}" failed. No retries permitted until 2026-07-30 17:05:38.189554344 +0000 UTC m=+312.676921728 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/fd8a1bec-7f8f-43ba-bce8-db1592ce379e-certificates") pod "keda-operator-77f648d694-7rshh" (UID: "fd8a1bec-7f8f-43ba-bce8-db1592ce379e") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Jul 30 17:05:37.689859 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:37.689840    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/fd8a1bec-7f8f-43ba-bce8-db1592ce379e-cabundle0\") pod \"keda-operator-77f648d694-7rshh\" (UID: \"fd8a1bec-7f8f-43ba-bce8-db1592ce379e\") " pod="openshift-keda/keda-operator-77f648d694-7rshh"
Jul 30 17:05:37.692634 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:37.692612    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-92bsq"]
Jul 30 17:05:37.696101 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:37.696081    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-92bsq"
Jul 30 17:05:37.698427 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:37.698410    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Jul 30 17:05:37.702419 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:37.702315    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn56t\" (UniqueName: \"kubernetes.io/projected/fd8a1bec-7f8f-43ba-bce8-db1592ce379e-kube-api-access-vn56t\") pod \"keda-operator-77f648d694-7rshh\" (UID: \"fd8a1bec-7f8f-43ba-bce8-db1592ce379e\") " pod="openshift-keda/keda-operator-77f648d694-7rshh"
Jul 30 17:05:37.704563 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:37.704542    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-92bsq"]
Jul 30 17:05:37.790024 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:37.789932    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/cceda4d4-98d8-4e66-a2f7-4011e30a8d1c-certificates\") pod \"keda-metrics-apiserver-55557df8c-92bsq\" (UID: \"cceda4d4-98d8-4e66-a2f7-4011e30a8d1c\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-92bsq"
Jul 30 17:05:37.790024 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:37.789963    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/cceda4d4-98d8-4e66-a2f7-4011e30a8d1c-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-92bsq\" (UID: \"cceda4d4-98d8-4e66-a2f7-4011e30a8d1c\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-92bsq"
Jul 30 17:05:37.790216 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:37.790069    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pn5m6\" (UniqueName: \"kubernetes.io/projected/cceda4d4-98d8-4e66-a2f7-4011e30a8d1c-kube-api-access-pn5m6\") pod \"keda-metrics-apiserver-55557df8c-92bsq\" (UID: \"cceda4d4-98d8-4e66-a2f7-4011e30a8d1c\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-92bsq"
Jul 30 17:05:37.824114 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:37.824078    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:05:37.883123 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:37.883088    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:05:37.891221 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:37.891199    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/cceda4d4-98d8-4e66-a2f7-4011e30a8d1c-certificates\") pod \"keda-metrics-apiserver-55557df8c-92bsq\" (UID: \"cceda4d4-98d8-4e66-a2f7-4011e30a8d1c\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-92bsq"
Jul 30 17:05:37.891306 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:37.891228    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/cceda4d4-98d8-4e66-a2f7-4011e30a8d1c-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-92bsq\" (UID: \"cceda4d4-98d8-4e66-a2f7-4011e30a8d1c\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-92bsq"
Jul 30 17:05:37.891306 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:37.891272    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn5m6\" (UniqueName: \"kubernetes.io/projected/cceda4d4-98d8-4e66-a2f7-4011e30a8d1c-kube-api-access-pn5m6\") pod \"keda-metrics-apiserver-55557df8c-92bsq\" (UID: \"cceda4d4-98d8-4e66-a2f7-4011e30a8d1c\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-92bsq"
Jul 30 17:05:37.891410 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:05:37.891369    2555 secret.go:281] references non-existent secret key: tls.crt
Jul 30 17:05:37.891410 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:05:37.891389    2555 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Jul 30 17:05:37.891499 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:05:37.891412    2555 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-55557df8c-92bsq: references non-existent secret key: tls.crt
Jul 30 17:05:37.891499 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:05:37.891476    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cceda4d4-98d8-4e66-a2f7-4011e30a8d1c-certificates podName:cceda4d4-98d8-4e66-a2f7-4011e30a8d1c nodeName:}" failed. No retries permitted until 2026-07-30 17:05:38.391456173 +0000 UTC m=+312.878823553 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/cceda4d4-98d8-4e66-a2f7-4011e30a8d1c-certificates") pod "keda-metrics-apiserver-55557df8c-92bsq" (UID: "cceda4d4-98d8-4e66-a2f7-4011e30a8d1c") : references non-existent secret key: tls.crt
Jul 30 17:05:37.891606 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:37.891543    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/cceda4d4-98d8-4e66-a2f7-4011e30a8d1c-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-92bsq\" (UID: \"cceda4d4-98d8-4e66-a2f7-4011e30a8d1c\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-92bsq"
Jul 30 17:05:37.903299 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:37.903272    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn5m6\" (UniqueName: \"kubernetes.io/projected/cceda4d4-98d8-4e66-a2f7-4011e30a8d1c-kube-api-access-pn5m6\") pod \"keda-metrics-apiserver-55557df8c-92bsq\" (UID: \"cceda4d4-98d8-4e66-a2f7-4011e30a8d1c\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-92bsq"
Jul 30 17:05:38.193612 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:38.193559    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/fd8a1bec-7f8f-43ba-bce8-db1592ce379e-certificates\") pod \"keda-operator-77f648d694-7rshh\" (UID: \"fd8a1bec-7f8f-43ba-bce8-db1592ce379e\") " pod="openshift-keda/keda-operator-77f648d694-7rshh"
Jul 30 17:05:38.193796 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:05:38.193721    2555 secret.go:281] references non-existent secret key: ca.crt
Jul 30 17:05:38.193796 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:05:38.193736    2555 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 30 17:05:38.193796 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:05:38.193745    2555 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-77f648d694-7rshh: references non-existent secret key: ca.crt
Jul 30 17:05:38.193902 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:05:38.193806    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd8a1bec-7f8f-43ba-bce8-db1592ce379e-certificates podName:fd8a1bec-7f8f-43ba-bce8-db1592ce379e nodeName:}" failed. No retries permitted until 2026-07-30 17:05:39.193790543 +0000 UTC m=+313.681157928 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/fd8a1bec-7f8f-43ba-bce8-db1592ce379e-certificates") pod "keda-operator-77f648d694-7rshh" (UID: "fd8a1bec-7f8f-43ba-bce8-db1592ce379e") : references non-existent secret key: ca.crt
Jul 30 17:05:38.377044 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:38.377002    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:05:38.394820 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:38.394792    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/cceda4d4-98d8-4e66-a2f7-4011e30a8d1c-certificates\") pod \"keda-metrics-apiserver-55557df8c-92bsq\" (UID: \"cceda4d4-98d8-4e66-a2f7-4011e30a8d1c\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-92bsq"
Jul 30 17:05:38.394908 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:05:38.394899    2555 secret.go:281] references non-existent secret key: tls.crt
Jul 30 17:05:38.394947 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:05:38.394912    2555 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Jul 30 17:05:38.394947 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:05:38.394928    2555 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-55557df8c-92bsq: references non-existent secret key: tls.crt
Jul 30 17:05:38.395027 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:05:38.394971    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cceda4d4-98d8-4e66-a2f7-4011e30a8d1c-certificates podName:cceda4d4-98d8-4e66-a2f7-4011e30a8d1c nodeName:}" failed. No retries permitted until 2026-07-30 17:05:39.39495964 +0000 UTC m=+313.882327020 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/cceda4d4-98d8-4e66-a2f7-4011e30a8d1c-certificates") pod "keda-metrics-apiserver-55557df8c-92bsq" (UID: "cceda4d4-98d8-4e66-a2f7-4011e30a8d1c") : references non-existent secret key: tls.crt
Jul 30 17:05:38.605452 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:38.605406    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-vrrxx"]
Jul 30 17:05:38.609351 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:38.609334    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-vrrxx"
Jul 30 17:05:38.612180 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:38.612159    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Jul 30 17:05:38.622257 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:38.622236    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-vrrxx"]
Jul 30 17:05:38.686486 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:38.686444    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:05:38.697646 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:38.697615    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4pl8\" (UniqueName: \"kubernetes.io/projected/12e27848-cce9-4519-a1b1-911740b4e172-kube-api-access-n4pl8\") pod \"keda-admission-56f9798bb4-vrrxx\" (UID: \"12e27848-cce9-4519-a1b1-911740b4e172\") " pod="openshift-keda/keda-admission-56f9798bb4-vrrxx"
Jul 30 17:05:38.697767 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:38.697686    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/12e27848-cce9-4519-a1b1-911740b4e172-certificates\") pod \"keda-admission-56f9798bb4-vrrxx\" (UID: \"12e27848-cce9-4519-a1b1-911740b4e172\") " pod="openshift-keda/keda-admission-56f9798bb4-vrrxx"
Jul 30 17:05:38.787289 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:38.787258    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:05:38.798568 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:38.798532    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/12e27848-cce9-4519-a1b1-911740b4e172-certificates\") pod \"keda-admission-56f9798bb4-vrrxx\" (UID: \"12e27848-cce9-4519-a1b1-911740b4e172\") " pod="openshift-keda/keda-admission-56f9798bb4-vrrxx"
Jul 30 17:05:38.798740 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:38.798693    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4pl8\" (UniqueName: \"kubernetes.io/projected/12e27848-cce9-4519-a1b1-911740b4e172-kube-api-access-n4pl8\") pod \"keda-admission-56f9798bb4-vrrxx\" (UID: \"12e27848-cce9-4519-a1b1-911740b4e172\") " pod="openshift-keda/keda-admission-56f9798bb4-vrrxx"
Jul 30 17:05:38.801067 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:38.801044    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/12e27848-cce9-4519-a1b1-911740b4e172-certificates\") pod \"keda-admission-56f9798bb4-vrrxx\" (UID: \"12e27848-cce9-4519-a1b1-911740b4e172\") " pod="openshift-keda/keda-admission-56f9798bb4-vrrxx"
Jul 30 17:05:38.812900 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:38.812870    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4pl8\" (UniqueName: \"kubernetes.io/projected/12e27848-cce9-4519-a1b1-911740b4e172-kube-api-access-n4pl8\") pod \"keda-admission-56f9798bb4-vrrxx\" (UID: \"12e27848-cce9-4519-a1b1-911740b4e172\") " pod="openshift-keda/keda-admission-56f9798bb4-vrrxx"
Jul 30 17:05:38.887324 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:38.887229    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:05:38.922668 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:38.922632    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-vrrxx"
Jul 30 17:05:39.073304 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:39.073281    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-vrrxx"]
Jul 30 17:05:39.076365 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:05:39.076326    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12e27848_cce9_4519_a1b1_911740b4e172.slice/crio-ded16df4ae16d5f92dddb0ddf417df303ca62645bbcd2e4a5c9266e6883e74f9 WatchSource:0}: Error finding container ded16df4ae16d5f92dddb0ddf417df303ca62645bbcd2e4a5c9266e6883e74f9: Status 404 returned error can't find the container with id ded16df4ae16d5f92dddb0ddf417df303ca62645bbcd2e4a5c9266e6883e74f9
Jul 30 17:05:39.077652 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:39.077622    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:05:39.204166 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:39.204082    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/fd8a1bec-7f8f-43ba-bce8-db1592ce379e-certificates\") pod \"keda-operator-77f648d694-7rshh\" (UID: \"fd8a1bec-7f8f-43ba-bce8-db1592ce379e\") " pod="openshift-keda/keda-operator-77f648d694-7rshh"
Jul 30 17:05:39.206546 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:39.206521    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/fd8a1bec-7f8f-43ba-bce8-db1592ce379e-certificates\") pod \"keda-operator-77f648d694-7rshh\" (UID: \"fd8a1bec-7f8f-43ba-bce8-db1592ce379e\") " pod="openshift-keda/keda-operator-77f648d694-7rshh"
Jul 30 17:05:39.277051 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:39.276994    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:05:39.297572 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:39.297538    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-7rshh"
Jul 30 17:05:39.379708 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:39.379682    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:05:39.406232 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:39.406202    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/cceda4d4-98d8-4e66-a2f7-4011e30a8d1c-certificates\") pod \"keda-metrics-apiserver-55557df8c-92bsq\" (UID: \"cceda4d4-98d8-4e66-a2f7-4011e30a8d1c\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-92bsq"
Jul 30 17:05:39.408763 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:39.408734    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/cceda4d4-98d8-4e66-a2f7-4011e30a8d1c-certificates\") pod \"keda-metrics-apiserver-55557df8c-92bsq\" (UID: \"cceda4d4-98d8-4e66-a2f7-4011e30a8d1c\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-92bsq"
Jul 30 17:05:39.432467 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:39.432438    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-7rshh"]
Jul 30 17:05:39.434870 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:05:39.434843    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd8a1bec_7f8f_43ba_bce8_db1592ce379e.slice/crio-9c73b90c9c44b1c3335f89e3702e0e7dbc60584f35c6d59b65726f488c6e04b3 WatchSource:0}: Error finding container 9c73b90c9c44b1c3335f89e3702e0e7dbc60584f35c6d59b65726f488c6e04b3: Status 404 returned error can't find the container with id 9c73b90c9c44b1c3335f89e3702e0e7dbc60584f35c6d59b65726f488c6e04b3
Jul 30 17:05:39.449260 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:39.449234    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:05:39.454990 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:39.454934    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-7rshh" event={"ID":"fd8a1bec-7f8f-43ba-bce8-db1592ce379e","Type":"ContainerStarted","Data":"9c73b90c9c44b1c3335f89e3702e0e7dbc60584f35c6d59b65726f488c6e04b3"}
Jul 30 17:05:39.456101 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:39.456078    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-vrrxx" event={"ID":"12e27848-cce9-4519-a1b1-911740b4e172","Type":"ContainerStarted","Data":"ded16df4ae16d5f92dddb0ddf417df303ca62645bbcd2e4a5c9266e6883e74f9"}
Jul 30 17:05:39.516813 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:39.516784    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-92bsq"
Jul 30 17:05:39.645655 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:39.645631    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-92bsq"]
Jul 30 17:05:39.647593 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:05:39.647564    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcceda4d4_98d8_4e66_a2f7_4011e30a8d1c.slice/crio-6fb21265061336049d95675b2d77dba5e8756b26c2566bd8bb3af58cbe13b089 WatchSource:0}: Error finding container 6fb21265061336049d95675b2d77dba5e8756b26c2566bd8bb3af58cbe13b089: Status 404 returned error can't find the container with id 6fb21265061336049d95675b2d77dba5e8756b26c2566bd8bb3af58cbe13b089
Jul 30 17:05:40.463929 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:40.463889    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-92bsq" event={"ID":"cceda4d4-98d8-4e66-a2f7-4011e30a8d1c","Type":"ContainerStarted","Data":"6fb21265061336049d95675b2d77dba5e8756b26c2566bd8bb3af58cbe13b089"}
Jul 30 17:05:41.474555 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:41.474489    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-vrrxx" event={"ID":"12e27848-cce9-4519-a1b1-911740b4e172","Type":"ContainerStarted","Data":"20a0e807834203149afcb892bc66c118b541f1cca1863c76e6a1b98e14fddfdd"}
Jul 30 17:05:41.475266 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:41.475245    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-56f9798bb4-vrrxx"
Jul 30 17:05:41.494465 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:41.494381    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-56f9798bb4-vrrxx" podStartSLOduration=1.910390763 podStartE2EDuration="3.494363121s" totalImagesPullingTime="1.583972358s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:05:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:05:41.493229227 +0000 UTC m=+315.980596630" watchObservedRunningTime="2026-07-30 17:05:41.494363121 +0000 UTC m=+315.981730523"
Jul 30 17:05:44.491389 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:44.491352    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-92bsq" event={"ID":"cceda4d4-98d8-4e66-a2f7-4011e30a8d1c","Type":"ContainerStarted","Data":"58cce4bc4d7feeb1217a3427082837a4384f1c052afd820b1dd60fa218ce32e9"}
Jul 30 17:05:44.491851 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:44.491533    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-92bsq"
Jul 30 17:05:44.492838 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:44.492817    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-7rshh" event={"ID":"fd8a1bec-7f8f-43ba-bce8-db1592ce379e","Type":"ContainerStarted","Data":"cf9a59ebd6f3363d5978d5a68ba6cfa8c6e9c9e116dfdbbe49c1e7943ca095bd"}
Jul 30 17:05:44.492956 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:44.492944    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-77f648d694-7rshh"
Jul 30 17:05:44.527249 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:44.527200    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-55557df8c-92bsq" podStartSLOduration=3.675877814 podStartE2EDuration="7.527185429s" totalImagesPullingTime="3.851307615s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:05:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:05:44.508040402 +0000 UTC m=+318.995407793" watchObservedRunningTime="2026-07-30 17:05:44.527185429 +0000 UTC m=+319.014552830"
Jul 30 17:05:44.527975 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:44.527953    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-77f648d694-7rshh" podStartSLOduration=3.464404813 podStartE2EDuration="7.527948447s" totalImagesPullingTime="4.063543634s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:05:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:05:44.526239069 +0000 UTC m=+319.013606471" watchObservedRunningTime="2026-07-30 17:05:44.527948447 +0000 UTC m=+319.015315888"
Jul 30 17:05:55.501964 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:55.501939    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-92bsq"
Jul 30 17:05:58.451456 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:05:58.451427    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-8wwx9"
Jul 30 17:06:03.487843 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:06:03.487810    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-56f9798bb4-vrrxx"
Jul 30 17:06:05.499740 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:06:05.499711    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-77f648d694-7rshh"
Jul 30 17:06:31.316631 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:06:31.316598    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:06:31.414691 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:06:31.414624    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:06:55.160174 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:06:55.160143    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:06:55.265395 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:06:55.265360    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-8677446b84-z7wqd"]
Jul 30 17:06:55.267868 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:06:55.267837    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-8677446b84-z7wqd"
Jul 30 17:06:55.268030 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:06:55.267966    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:06:55.270693 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:06:55.270676    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 30 17:06:55.270770 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:06:55.270716    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 30 17:06:55.270770 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:06:55.270678    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-j68z8\""
Jul 30 17:06:55.271799 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:06:55.271784    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 30 17:06:55.304457 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:06:55.304427    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-8677446b84-z7wqd"]
Jul 30 17:06:55.338034 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:06:55.337779    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-jv5m4"]
Jul 30 17:06:55.340659 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:06:55.340634    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-jv5m4"
Jul 30 17:06:55.344043 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:06:55.343999    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 30 17:06:55.344197 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:06:55.344048    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-cmd5r\""
Jul 30 17:06:55.345555 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:06:55.345531    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn95p\" (UniqueName: \"kubernetes.io/projected/ca5cdf77-1358-4231-82b5-c34bc619d909-kube-api-access-rn95p\") pod \"kserve-controller-manager-8677446b84-z7wqd\" (UID: \"ca5cdf77-1358-4231-82b5-c34bc619d909\") " pod="kserve/kserve-controller-manager-8677446b84-z7wqd"
Jul 30 17:06:55.345827 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:06:55.345798    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ca5cdf77-1358-4231-82b5-c34bc619d909-cert\") pod \"kserve-controller-manager-8677446b84-z7wqd\" (UID: \"ca5cdf77-1358-4231-82b5-c34bc619d909\") " pod="kserve/kserve-controller-manager-8677446b84-z7wqd"
Jul 30 17:06:55.346082 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:06:55.346054    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:06:55.347949 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:06:55.347926    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-jv5m4"]
Jul 30 17:06:55.437911 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:06:55.437817    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:06:55.446726 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:06:55.446702    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d4e163c0-ecd2-467e-b228-5a97deba9f0e-data\") pod \"seaweedfs-64568bcd49-jv5m4\" (UID: \"d4e163c0-ecd2-467e-b228-5a97deba9f0e\") " pod="kserve/seaweedfs-64568bcd49-jv5m4"
Jul 30 17:06:55.446805 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:06:55.446734    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhpl2\" (UniqueName: \"kubernetes.io/projected/d4e163c0-ecd2-467e-b228-5a97deba9f0e-kube-api-access-dhpl2\") pod \"seaweedfs-64568bcd49-jv5m4\" (UID: \"d4e163c0-ecd2-467e-b228-5a97deba9f0e\") " pod="kserve/seaweedfs-64568bcd49-jv5m4"
Jul 30 17:06:55.446805 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:06:55.446785    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn95p\" (UniqueName: \"kubernetes.io/projected/ca5cdf77-1358-4231-82b5-c34bc619d909-kube-api-access-rn95p\") pod \"kserve-controller-manager-8677446b84-z7wqd\" (UID: \"ca5cdf77-1358-4231-82b5-c34bc619d909\") " pod="kserve/kserve-controller-manager-8677446b84-z7wqd"
Jul 30 17:06:55.446893 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:06:55.446875    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ca5cdf77-1358-4231-82b5-c34bc619d909-cert\") pod \"kserve-controller-manager-8677446b84-z7wqd\" (UID: \"ca5cdf77-1358-4231-82b5-c34bc619d909\") " pod="kserve/kserve-controller-manager-8677446b84-z7wqd"
Jul 30 17:06:55.446992 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:06:55.446978    2555 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Jul 30 17:06:55.447078 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:06:55.447068    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca5cdf77-1358-4231-82b5-c34bc619d909-cert podName:ca5cdf77-1358-4231-82b5-c34bc619d909 nodeName:}" failed. No retries permitted until 2026-07-30 17:06:55.947047703 +0000 UTC m=+390.434415085 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/ca5cdf77-1358-4231-82b5-c34bc619d909-cert") pod "kserve-controller-manager-8677446b84-z7wqd" (UID: "ca5cdf77-1358-4231-82b5-c34bc619d909") : secret "kserve-webhook-server-cert" not found
Jul 30 17:06:55.460828 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:06:55.460798    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn95p\" (UniqueName: \"kubernetes.io/projected/ca5cdf77-1358-4231-82b5-c34bc619d909-kube-api-access-rn95p\") pod \"kserve-controller-manager-8677446b84-z7wqd\" (UID: \"ca5cdf77-1358-4231-82b5-c34bc619d909\") " pod="kserve/kserve-controller-manager-8677446b84-z7wqd"
Jul 30 17:06:55.547941 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:06:55.547899    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d4e163c0-ecd2-467e-b228-5a97deba9f0e-data\") pod \"seaweedfs-64568bcd49-jv5m4\" (UID: \"d4e163c0-ecd2-467e-b228-5a97deba9f0e\") " pod="kserve/seaweedfs-64568bcd49-jv5m4"
Jul 30 17:06:55.547941 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:06:55.547937    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhpl2\" (UniqueName: \"kubernetes.io/projected/d4e163c0-ecd2-467e-b228-5a97deba9f0e-kube-api-access-dhpl2\") pod \"seaweedfs-64568bcd49-jv5m4\" (UID: \"d4e163c0-ecd2-467e-b228-5a97deba9f0e\") " pod="kserve/seaweedfs-64568bcd49-jv5m4"
Jul 30 17:06:55.548365 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:06:55.548347    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d4e163c0-ecd2-467e-b228-5a97deba9f0e-data\") pod \"seaweedfs-64568bcd49-jv5m4\" (UID: \"d4e163c0-ecd2-467e-b228-5a97deba9f0e\") " pod="kserve/seaweedfs-64568bcd49-jv5m4"
Jul 30 17:06:55.558681 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:06:55.558662    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhpl2\" (UniqueName: \"kubernetes.io/projected/d4e163c0-ecd2-467e-b228-5a97deba9f0e-kube-api-access-dhpl2\") pod \"seaweedfs-64568bcd49-jv5m4\" (UID: \"d4e163c0-ecd2-467e-b228-5a97deba9f0e\") " pod="kserve/seaweedfs-64568bcd49-jv5m4"
Jul 30 17:06:55.652851 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:06:55.652810    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-jv5m4"
Jul 30 17:06:55.705620 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:06:55.705591    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:06:55.787994 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:06:55.787959    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-jv5m4"]
Jul 30 17:06:55.791135 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:06:55.791113    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4e163c0_ecd2_467e_b228_5a97deba9f0e.slice/crio-ea59e6078df1640f60d704c586edd8dd0c0ff988e53948a3df51c28df7a19f3a WatchSource:0}: Error finding container ea59e6078df1640f60d704c586edd8dd0c0ff988e53948a3df51c28df7a19f3a: Status 404 returned error can't find the container with id ea59e6078df1640f60d704c586edd8dd0c0ff988e53948a3df51c28df7a19f3a
Jul 30 17:06:55.807692 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:06:55.807665    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-jv5m4" event={"ID":"d4e163c0-ecd2-467e-b228-5a97deba9f0e","Type":"ContainerStarted","Data":"ea59e6078df1640f60d704c586edd8dd0c0ff988e53948a3df51c28df7a19f3a"}
Jul 30 17:06:55.846717 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:06:55.846688    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:06:55.947730 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:06:55.947698    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:06:55.951416 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:06:55.951393    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ca5cdf77-1358-4231-82b5-c34bc619d909-cert\") pod \"kserve-controller-manager-8677446b84-z7wqd\" (UID: \"ca5cdf77-1358-4231-82b5-c34bc619d909\") " pod="kserve/kserve-controller-manager-8677446b84-z7wqd"
Jul 30 17:06:55.953737 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:06:55.953684    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ca5cdf77-1358-4231-82b5-c34bc619d909-cert\") pod \"kserve-controller-manager-8677446b84-z7wqd\" (UID: \"ca5cdf77-1358-4231-82b5-c34bc619d909\") " pod="kserve/kserve-controller-manager-8677446b84-z7wqd"
Jul 30 17:06:56.050045 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:06:56.050001    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:06:56.116889 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:06:56.116859    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:06:56.178311 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:06:56.178280    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-8677446b84-z7wqd"
Jul 30 17:06:56.324416 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:06:56.324394    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-8677446b84-z7wqd"]
Jul 30 17:06:56.326226 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:06:56.326202    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca5cdf77_1358_4231_82b5_c34bc619d909.slice/crio-adbebb75f998e0995df027467ff83745789da436c12e155de34488f2bbb34f3c WatchSource:0}: Error finding container adbebb75f998e0995df027467ff83745789da436c12e155de34488f2bbb34f3c: Status 404 returned error can't find the container with id adbebb75f998e0995df027467ff83745789da436c12e155de34488f2bbb34f3c
Jul 30 17:06:56.814320 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:06:56.814284    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-8677446b84-z7wqd" event={"ID":"ca5cdf77-1358-4231-82b5-c34bc619d909","Type":"ContainerStarted","Data":"adbebb75f998e0995df027467ff83745789da436c12e155de34488f2bbb34f3c"}
Jul 30 17:07:00.836080 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:07:00.836045    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-jv5m4" event={"ID":"d4e163c0-ecd2-467e-b228-5a97deba9f0e","Type":"ContainerStarted","Data":"12f4d34ad76afb0d6f3dc05860314604f2f75d4b10ad027bd0e8972eed10b701"}
Jul 30 17:07:00.836628 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:07:00.836170    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-jv5m4"
Jul 30 17:07:00.837505 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:07:00.837483    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-8677446b84-z7wqd" event={"ID":"ca5cdf77-1358-4231-82b5-c34bc619d909","Type":"ContainerStarted","Data":"75a05e8d4a0aebbac45355116d10daee8e64d8e348a0b453d5698f4bcabfd0f7"}
Jul 30 17:07:00.837627 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:07:00.837617    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-8677446b84-z7wqd"
Jul 30 17:07:00.854518 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:07:00.854479    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-jv5m4" podStartSLOduration=1.7879704090000001 podStartE2EDuration="5.854466809s" totalImagesPullingTime="4.0664964s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:06:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:07:00.853205329 +0000 UTC m=+395.340572729" watchObservedRunningTime="2026-07-30 17:07:00.854466809 +0000 UTC m=+395.341834209"
Jul 30 17:07:00.870904 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:07:00.870865    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-8677446b84-z7wqd" podStartSLOduration=2.426196108 podStartE2EDuration="5.87085416s" totalImagesPullingTime="3.444658052s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:06:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:07:00.869389312 +0000 UTC m=+395.356756726" watchObservedRunningTime="2026-07-30 17:07:00.87085416 +0000 UTC m=+395.358221561"
Jul 30 17:07:06.843423 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:07:06.843389    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-jv5m4"
Jul 30 17:07:08.560398 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:07:08.560369    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:07:31.846639 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:07:31.846610    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-8677446b84-z7wqd"
Jul 30 17:07:34.386285 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:07:34.386254    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:07:37.261543 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:07:37.261510    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-8677446b84-z7wqd"]
Jul 30 17:07:37.261944 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:07:37.261697    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-8677446b84-z7wqd" podUID="ca5cdf77-1358-4231-82b5-c34bc619d909" containerName="manager" containerID="cri-o://75a05e8d4a0aebbac45355116d10daee8e64d8e348a0b453d5698f4bcabfd0f7" gracePeriod=10
Jul 30 17:07:37.291453 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:07:37.291427    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-8677446b84-rz8nz"]
Jul 30 17:07:37.293860 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:07:37.293845    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-8677446b84-rz8nz"
Jul 30 17:07:37.305935 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:07:37.305908    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-8677446b84-rz8nz"]
Jul 30 17:07:37.385887 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:07:37.385855    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nwgf\" (UniqueName: \"kubernetes.io/projected/6ed65973-197a-4225-b161-4f847067a7c1-kube-api-access-4nwgf\") pod \"kserve-controller-manager-8677446b84-rz8nz\" (UID: \"6ed65973-197a-4225-b161-4f847067a7c1\") " pod="kserve/kserve-controller-manager-8677446b84-rz8nz"
Jul 30 17:07:37.386048 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:07:37.385903    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6ed65973-197a-4225-b161-4f847067a7c1-cert\") pod \"kserve-controller-manager-8677446b84-rz8nz\" (UID: \"6ed65973-197a-4225-b161-4f847067a7c1\") " pod="kserve/kserve-controller-manager-8677446b84-rz8nz"
Jul 30 17:07:37.486600 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:07:37.486573    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nwgf\" (UniqueName: \"kubernetes.io/projected/6ed65973-197a-4225-b161-4f847067a7c1-kube-api-access-4nwgf\") pod \"kserve-controller-manager-8677446b84-rz8nz\" (UID: \"6ed65973-197a-4225-b161-4f847067a7c1\") " pod="kserve/kserve-controller-manager-8677446b84-rz8nz"
Jul 30 17:07:37.486703 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:07:37.486612    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6ed65973-197a-4225-b161-4f847067a7c1-cert\") pod \"kserve-controller-manager-8677446b84-rz8nz\" (UID: \"6ed65973-197a-4225-b161-4f847067a7c1\") " pod="kserve/kserve-controller-manager-8677446b84-rz8nz"
Jul 30 17:07:37.489031 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:07:37.488995    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6ed65973-197a-4225-b161-4f847067a7c1-cert\") pod \"kserve-controller-manager-8677446b84-rz8nz\" (UID: \"6ed65973-197a-4225-b161-4f847067a7c1\") " pod="kserve/kserve-controller-manager-8677446b84-rz8nz"
Jul 30 17:07:37.496812 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:07:37.496790    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nwgf\" (UniqueName: \"kubernetes.io/projected/6ed65973-197a-4225-b161-4f847067a7c1-kube-api-access-4nwgf\") pod \"kserve-controller-manager-8677446b84-rz8nz\" (UID: \"6ed65973-197a-4225-b161-4f847067a7c1\") " pod="kserve/kserve-controller-manager-8677446b84-rz8nz"
Jul 30 17:07:37.507016 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:07:37.506987    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-8677446b84-z7wqd"
Jul 30 17:07:37.587415 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:07:37.587373    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ca5cdf77-1358-4231-82b5-c34bc619d909-cert\") pod \"ca5cdf77-1358-4231-82b5-c34bc619d909\" (UID: \"ca5cdf77-1358-4231-82b5-c34bc619d909\") "
Jul 30 17:07:37.587610 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:07:37.587495    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rn95p\" (UniqueName: \"kubernetes.io/projected/ca5cdf77-1358-4231-82b5-c34bc619d909-kube-api-access-rn95p\") pod \"ca5cdf77-1358-4231-82b5-c34bc619d909\" (UID: \"ca5cdf77-1358-4231-82b5-c34bc619d909\") "
Jul 30 17:07:37.589490 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:07:37.589464    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca5cdf77-1358-4231-82b5-c34bc619d909-cert" (OuterVolumeSpecName: "cert") pod "ca5cdf77-1358-4231-82b5-c34bc619d909" (UID: "ca5cdf77-1358-4231-82b5-c34bc619d909"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:07:37.589592 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:07:37.589515    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca5cdf77-1358-4231-82b5-c34bc619d909-kube-api-access-rn95p" (OuterVolumeSpecName: "kube-api-access-rn95p") pod "ca5cdf77-1358-4231-82b5-c34bc619d909" (UID: "ca5cdf77-1358-4231-82b5-c34bc619d909"). InnerVolumeSpecName "kube-api-access-rn95p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 17:07:37.653870 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:07:37.653835    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-8677446b84-rz8nz"
Jul 30 17:07:37.688813 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:07:37.688775    2555 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ca5cdf77-1358-4231-82b5-c34bc619d909-cert\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:07:37.688813 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:07:37.688800    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rn95p\" (UniqueName: \"kubernetes.io/projected/ca5cdf77-1358-4231-82b5-c34bc619d909-kube-api-access-rn95p\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:07:37.779867 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:07:37.779845    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-8677446b84-rz8nz"]
Jul 30 17:07:37.781493 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:07:37.781466    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ed65973_197a_4225_b161_4f847067a7c1.slice/crio-4a1419f1d70c3faa1e3bf19148ccba38896d9d1c39be5d3d5e64846da8bb3f70 WatchSource:0}: Error finding container 4a1419f1d70c3faa1e3bf19148ccba38896d9d1c39be5d3d5e64846da8bb3f70: Status 404 returned error can't find the container with id 4a1419f1d70c3faa1e3bf19148ccba38896d9d1c39be5d3d5e64846da8bb3f70
Jul 30 17:07:37.997899 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:07:37.997816    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-8677446b84-rz8nz" event={"ID":"6ed65973-197a-4225-b161-4f847067a7c1","Type":"ContainerStarted","Data":"4a1419f1d70c3faa1e3bf19148ccba38896d9d1c39be5d3d5e64846da8bb3f70"}
Jul 30 17:07:37.999029 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:07:37.998990    2555 generic.go:356] "Generic (PLEG): container finished" podID="ca5cdf77-1358-4231-82b5-c34bc619d909" containerID="75a05e8d4a0aebbac45355116d10daee8e64d8e348a0b453d5698f4bcabfd0f7" exitCode=0
Jul 30 17:07:37.999148 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:07:37.999064    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-8677446b84-z7wqd"
Jul 30 17:07:37.999148 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:07:37.999076    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-8677446b84-z7wqd" event={"ID":"ca5cdf77-1358-4231-82b5-c34bc619d909","Type":"ContainerDied","Data":"75a05e8d4a0aebbac45355116d10daee8e64d8e348a0b453d5698f4bcabfd0f7"}
Jul 30 17:07:37.999148 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:07:37.999126    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-8677446b84-z7wqd" event={"ID":"ca5cdf77-1358-4231-82b5-c34bc619d909","Type":"ContainerDied","Data":"adbebb75f998e0995df027467ff83745789da436c12e155de34488f2bbb34f3c"}
Jul 30 17:07:37.999148 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:07:37.999148    2555 scope.go:117] "RemoveContainer" containerID="75a05e8d4a0aebbac45355116d10daee8e64d8e348a0b453d5698f4bcabfd0f7"
Jul 30 17:07:38.007552 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:07:38.007537    2555 scope.go:117] "RemoveContainer" containerID="75a05e8d4a0aebbac45355116d10daee8e64d8e348a0b453d5698f4bcabfd0f7"
Jul 30 17:07:38.007799 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:07:38.007784    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75a05e8d4a0aebbac45355116d10daee8e64d8e348a0b453d5698f4bcabfd0f7\": container with ID starting with 75a05e8d4a0aebbac45355116d10daee8e64d8e348a0b453d5698f4bcabfd0f7 not found: ID does not exist" containerID="75a05e8d4a0aebbac45355116d10daee8e64d8e348a0b453d5698f4bcabfd0f7"
Jul 30 17:07:38.007852 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:07:38.007808    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75a05e8d4a0aebbac45355116d10daee8e64d8e348a0b453d5698f4bcabfd0f7"} err="failed to get container status \"75a05e8d4a0aebbac45355116d10daee8e64d8e348a0b453d5698f4bcabfd0f7\": rpc error: code = NotFound desc = could not find container \"75a05e8d4a0aebbac45355116d10daee8e64d8e348a0b453d5698f4bcabfd0f7\": container with ID starting with 75a05e8d4a0aebbac45355116d10daee8e64d8e348a0b453d5698f4bcabfd0f7 not found: ID does not exist"
Jul 30 17:07:38.021606 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:07:38.021581    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-8677446b84-z7wqd"]
Jul 30 17:07:38.029659 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:07:38.029636    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-8677446b84-z7wqd"]
Jul 30 17:07:38.060612 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:07:38.060584    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca5cdf77-1358-4231-82b5-c34bc619d909" path="/var/lib/kubelet/pods/ca5cdf77-1358-4231-82b5-c34bc619d909/volumes"
Jul 30 17:07:39.004669 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:07:39.004636    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-8677446b84-rz8nz" event={"ID":"6ed65973-197a-4225-b161-4f847067a7c1","Type":"ContainerStarted","Data":"72883ba9cb6e6a4bd5f71638230f1ac3f2f583aa4b4d359a90c2287b8f42182e"}
Jul 30 17:07:39.005128 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:07:39.004756    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-8677446b84-rz8nz"
Jul 30 17:07:39.021969 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:07:39.021923    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-8677446b84-rz8nz" podStartSLOduration=1.527711764 podStartE2EDuration="2.02190994s" totalImagesPullingTime="494.198176ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:07:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:07:39.021766307 +0000 UTC m=+433.509133708" watchObservedRunningTime="2026-07-30 17:07:39.02190994 +0000 UTC m=+433.509277340"
Jul 30 17:07:43.376671 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:07:43.376641    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:07:45.504769 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:07:45.504734    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:07:45.605219 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:07:45.605189    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:08:04.846568 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:04.846494    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-857dfd46db-84ghf"]
Jul 30 17:08:04.846936 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:04.846821    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca5cdf77-1358-4231-82b5-c34bc619d909" containerName="manager"
Jul 30 17:08:04.846936 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:04.846832    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca5cdf77-1358-4231-82b5-c34bc619d909" containerName="manager"
Jul 30 17:08:04.846936 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:04.846888    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca5cdf77-1358-4231-82b5-c34bc619d909" containerName="manager"
Jul 30 17:08:04.852000 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:04.851985    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-857dfd46db-84ghf"
Jul 30 17:08:04.865643 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:04.865617    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-857dfd46db-84ghf"]
Jul 30 17:08:04.926628 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:04.926595    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/18d3d3a8-db9c-4a25-bf94-535f65ad2ea6-console-oauth-config\") pod \"console-857dfd46db-84ghf\" (UID: \"18d3d3a8-db9c-4a25-bf94-535f65ad2ea6\") " pod="openshift-console/console-857dfd46db-84ghf"
Jul 30 17:08:04.926798 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:04.926642    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxp9p\" (UniqueName: \"kubernetes.io/projected/18d3d3a8-db9c-4a25-bf94-535f65ad2ea6-kube-api-access-qxp9p\") pod \"console-857dfd46db-84ghf\" (UID: \"18d3d3a8-db9c-4a25-bf94-535f65ad2ea6\") " pod="openshift-console/console-857dfd46db-84ghf"
Jul 30 17:08:04.926798 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:04.926700    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/18d3d3a8-db9c-4a25-bf94-535f65ad2ea6-oauth-serving-cert\") pod \"console-857dfd46db-84ghf\" (UID: \"18d3d3a8-db9c-4a25-bf94-535f65ad2ea6\") " pod="openshift-console/console-857dfd46db-84ghf"
Jul 30 17:08:04.926798 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:04.926743    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/18d3d3a8-db9c-4a25-bf94-535f65ad2ea6-service-ca\") pod \"console-857dfd46db-84ghf\" (UID: \"18d3d3a8-db9c-4a25-bf94-535f65ad2ea6\") " pod="openshift-console/console-857dfd46db-84ghf"
Jul 30 17:08:04.926798 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:04.926773    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/18d3d3a8-db9c-4a25-bf94-535f65ad2ea6-console-config\") pod \"console-857dfd46db-84ghf\" (UID: \"18d3d3a8-db9c-4a25-bf94-535f65ad2ea6\") " pod="openshift-console/console-857dfd46db-84ghf"
Jul 30 17:08:04.926970 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:04.926801    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18d3d3a8-db9c-4a25-bf94-535f65ad2ea6-trusted-ca-bundle\") pod \"console-857dfd46db-84ghf\" (UID: \"18d3d3a8-db9c-4a25-bf94-535f65ad2ea6\") " pod="openshift-console/console-857dfd46db-84ghf"
Jul 30 17:08:04.926970 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:04.926830    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/18d3d3a8-db9c-4a25-bf94-535f65ad2ea6-console-serving-cert\") pod \"console-857dfd46db-84ghf\" (UID: \"18d3d3a8-db9c-4a25-bf94-535f65ad2ea6\") " pod="openshift-console/console-857dfd46db-84ghf"
Jul 30 17:08:04.927069 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:04.926999    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:08:04.995907 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:04.995875    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:08:05.027284 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:05.027252    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/18d3d3a8-db9c-4a25-bf94-535f65ad2ea6-service-ca\") pod \"console-857dfd46db-84ghf\" (UID: \"18d3d3a8-db9c-4a25-bf94-535f65ad2ea6\") " pod="openshift-console/console-857dfd46db-84ghf"
Jul 30 17:08:05.027435 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:05.027290    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/18d3d3a8-db9c-4a25-bf94-535f65ad2ea6-console-config\") pod \"console-857dfd46db-84ghf\" (UID: \"18d3d3a8-db9c-4a25-bf94-535f65ad2ea6\") " pod="openshift-console/console-857dfd46db-84ghf"
Jul 30 17:08:05.027435 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:05.027318    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18d3d3a8-db9c-4a25-bf94-535f65ad2ea6-trusted-ca-bundle\") pod \"console-857dfd46db-84ghf\" (UID: \"18d3d3a8-db9c-4a25-bf94-535f65ad2ea6\") " pod="openshift-console/console-857dfd46db-84ghf"
Jul 30 17:08:05.027435 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:05.027346    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/18d3d3a8-db9c-4a25-bf94-535f65ad2ea6-console-serving-cert\") pod \"console-857dfd46db-84ghf\" (UID: \"18d3d3a8-db9c-4a25-bf94-535f65ad2ea6\") " pod="openshift-console/console-857dfd46db-84ghf"
Jul 30 17:08:05.027435 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:05.027414    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/18d3d3a8-db9c-4a25-bf94-535f65ad2ea6-console-oauth-config\") pod \"console-857dfd46db-84ghf\" (UID: \"18d3d3a8-db9c-4a25-bf94-535f65ad2ea6\") " pod="openshift-console/console-857dfd46db-84ghf"
Jul 30 17:08:05.027623 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:05.027444    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxp9p\" (UniqueName: \"kubernetes.io/projected/18d3d3a8-db9c-4a25-bf94-535f65ad2ea6-kube-api-access-qxp9p\") pod \"console-857dfd46db-84ghf\" (UID: \"18d3d3a8-db9c-4a25-bf94-535f65ad2ea6\") " pod="openshift-console/console-857dfd46db-84ghf"
Jul 30 17:08:05.027623 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:05.027489    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/18d3d3a8-db9c-4a25-bf94-535f65ad2ea6-oauth-serving-cert\") pod \"console-857dfd46db-84ghf\" (UID: \"18d3d3a8-db9c-4a25-bf94-535f65ad2ea6\") " pod="openshift-console/console-857dfd46db-84ghf"
Jul 30 17:08:05.028147 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:05.028118    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/18d3d3a8-db9c-4a25-bf94-535f65ad2ea6-service-ca\") pod \"console-857dfd46db-84ghf\" (UID: \"18d3d3a8-db9c-4a25-bf94-535f65ad2ea6\") " pod="openshift-console/console-857dfd46db-84ghf"
Jul 30 17:08:05.028147 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:05.028138    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/18d3d3a8-db9c-4a25-bf94-535f65ad2ea6-console-config\") pod \"console-857dfd46db-84ghf\" (UID: \"18d3d3a8-db9c-4a25-bf94-535f65ad2ea6\") " pod="openshift-console/console-857dfd46db-84ghf"
Jul 30 17:08:05.028344 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:05.028202    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18d3d3a8-db9c-4a25-bf94-535f65ad2ea6-trusted-ca-bundle\") pod \"console-857dfd46db-84ghf\" (UID: \"18d3d3a8-db9c-4a25-bf94-535f65ad2ea6\") " pod="openshift-console/console-857dfd46db-84ghf"
Jul 30 17:08:05.028344 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:05.028291    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/18d3d3a8-db9c-4a25-bf94-535f65ad2ea6-oauth-serving-cert\") pod \"console-857dfd46db-84ghf\" (UID: \"18d3d3a8-db9c-4a25-bf94-535f65ad2ea6\") " pod="openshift-console/console-857dfd46db-84ghf"
Jul 30 17:08:05.030324 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:05.030308    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/18d3d3a8-db9c-4a25-bf94-535f65ad2ea6-console-serving-cert\") pod \"console-857dfd46db-84ghf\" (UID: \"18d3d3a8-db9c-4a25-bf94-535f65ad2ea6\") " pod="openshift-console/console-857dfd46db-84ghf"
Jul 30 17:08:05.030436 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:05.030412    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/18d3d3a8-db9c-4a25-bf94-535f65ad2ea6-console-oauth-config\") pod \"console-857dfd46db-84ghf\" (UID: \"18d3d3a8-db9c-4a25-bf94-535f65ad2ea6\") " pod="openshift-console/console-857dfd46db-84ghf"
Jul 30 17:08:05.037394 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:05.037372    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxp9p\" (UniqueName: \"kubernetes.io/projected/18d3d3a8-db9c-4a25-bf94-535f65ad2ea6-kube-api-access-qxp9p\") pod \"console-857dfd46db-84ghf\" (UID: \"18d3d3a8-db9c-4a25-bf94-535f65ad2ea6\") " pod="openshift-console/console-857dfd46db-84ghf"
Jul 30 17:08:05.160795 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:05.160716    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-857dfd46db-84ghf"
Jul 30 17:08:05.297060 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:05.297034    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-857dfd46db-84ghf"]
Jul 30 17:08:05.299173 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:08:05.299148    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18d3d3a8_db9c_4a25_bf94_535f65ad2ea6.slice/crio-688ee10d38e4fd26898f8b247f0b445cc513a67497b6b1b216319552fb3a1f25 WatchSource:0}: Error finding container 688ee10d38e4fd26898f8b247f0b445cc513a67497b6b1b216319552fb3a1f25: Status 404 returned error can't find the container with id 688ee10d38e4fd26898f8b247f0b445cc513a67497b6b1b216319552fb3a1f25
Jul 30 17:08:06.123182 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:06.123141    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-857dfd46db-84ghf" event={"ID":"18d3d3a8-db9c-4a25-bf94-535f65ad2ea6","Type":"ContainerStarted","Data":"de2e2119dd38970b49bb537c86fe2ff43a1de5166c0a3dac9535083d232e5ea9"}
Jul 30 17:08:06.123635 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:06.123198    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-857dfd46db-84ghf" event={"ID":"18d3d3a8-db9c-4a25-bf94-535f65ad2ea6","Type":"ContainerStarted","Data":"688ee10d38e4fd26898f8b247f0b445cc513a67497b6b1b216319552fb3a1f25"}
Jul 30 17:08:06.145761 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:06.145709    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-857dfd46db-84ghf" podStartSLOduration=2.145692065 podStartE2EDuration="2.145692065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:08:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:08:06.144662885 +0000 UTC m=+460.632030287" watchObservedRunningTime="2026-07-30 17:08:06.145692065 +0000 UTC m=+460.633059465"
Jul 30 17:08:10.014963 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:10.014926    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-8677446b84-rz8nz"
Jul 30 17:08:11.037177 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:11.037147    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:08:11.080858 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:11.080825    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-kvr8r"]
Jul 30 17:08:11.084404 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:11.084383    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-kvr8r"
Jul 30 17:08:11.086895 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:11.086871    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 30 17:08:11.086895 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:11.086893    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-4gcrb\""
Jul 30 17:08:11.093442 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:11.093417    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-kvr8r"]
Jul 30 17:08:11.098035 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:11.098015    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-8588677cfb-zbjs8"]
Jul 30 17:08:11.101258 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:11.101242    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-8588677cfb-zbjs8"
Jul 30 17:08:11.103762 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:11.103744    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Jul 30 17:08:11.103883 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:11.103769    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-k92fp\""
Jul 30 17:08:11.103883 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:11.103812    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 30 17:08:11.104102 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:11.104077    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:08:11.109939 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:11.109914    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-8588677cfb-zbjs8"]
Jul 30 17:08:11.168600 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:11.168569    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:08:11.170260 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:11.170243    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/79f9bec6-db1a-4830-b663-2f3435132f67-cert\") pod \"odh-model-controller-8588677cfb-zbjs8\" (UID: \"79f9bec6-db1a-4830-b663-2f3435132f67\") " pod="kserve/odh-model-controller-8588677cfb-zbjs8"
Jul 30 17:08:11.170333 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:11.170273    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79f9bec6-db1a-4830-b663-2f3435132f67-metrics-certs\") pod \"odh-model-controller-8588677cfb-zbjs8\" (UID: \"79f9bec6-db1a-4830-b663-2f3435132f67\") " pod="kserve/odh-model-controller-8588677cfb-zbjs8"
Jul 30 17:08:11.170333 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:11.170291    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw8jz\" (UniqueName: \"kubernetes.io/projected/79f9bec6-db1a-4830-b663-2f3435132f67-kube-api-access-hw8jz\") pod \"odh-model-controller-8588677cfb-zbjs8\" (UID: \"79f9bec6-db1a-4830-b663-2f3435132f67\") " pod="kserve/odh-model-controller-8588677cfb-zbjs8"
Jul 30 17:08:11.170436 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:11.170406    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj29b\" (UniqueName: \"kubernetes.io/projected/0d30e976-fa6b-44af-9a2a-5d8433a3cd84-kube-api-access-pj29b\") pod \"model-serving-api-7b954ffc95-kvr8r\" (UID: \"0d30e976-fa6b-44af-9a2a-5d8433a3cd84\") " pod="kserve/model-serving-api-7b954ffc95-kvr8r"
Jul 30 17:08:11.170486 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:11.170450    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d30e976-fa6b-44af-9a2a-5d8433a3cd84-tls-certs\") pod \"model-serving-api-7b954ffc95-kvr8r\" (UID: \"0d30e976-fa6b-44af-9a2a-5d8433a3cd84\") " pod="kserve/model-serving-api-7b954ffc95-kvr8r"
Jul 30 17:08:11.270936 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:11.270904    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj29b\" (UniqueName: \"kubernetes.io/projected/0d30e976-fa6b-44af-9a2a-5d8433a3cd84-kube-api-access-pj29b\") pod \"model-serving-api-7b954ffc95-kvr8r\" (UID: \"0d30e976-fa6b-44af-9a2a-5d8433a3cd84\") " pod="kserve/model-serving-api-7b954ffc95-kvr8r"
Jul 30 17:08:11.271089 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:11.270943    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d30e976-fa6b-44af-9a2a-5d8433a3cd84-tls-certs\") pod \"model-serving-api-7b954ffc95-kvr8r\" (UID: \"0d30e976-fa6b-44af-9a2a-5d8433a3cd84\") " pod="kserve/model-serving-api-7b954ffc95-kvr8r"
Jul 30 17:08:11.271089 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:11.270965    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/79f9bec6-db1a-4830-b663-2f3435132f67-cert\") pod \"odh-model-controller-8588677cfb-zbjs8\" (UID: \"79f9bec6-db1a-4830-b663-2f3435132f67\") " pod="kserve/odh-model-controller-8588677cfb-zbjs8"
Jul 30 17:08:11.271089 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:11.270992    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79f9bec6-db1a-4830-b663-2f3435132f67-metrics-certs\") pod \"odh-model-controller-8588677cfb-zbjs8\" (UID: \"79f9bec6-db1a-4830-b663-2f3435132f67\") " pod="kserve/odh-model-controller-8588677cfb-zbjs8"
Jul 30 17:08:11.271299 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:11.271167    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw8jz\" (UniqueName: \"kubernetes.io/projected/79f9bec6-db1a-4830-b663-2f3435132f67-kube-api-access-hw8jz\") pod \"odh-model-controller-8588677cfb-zbjs8\" (UID: \"79f9bec6-db1a-4830-b663-2f3435132f67\") " pod="kserve/odh-model-controller-8588677cfb-zbjs8"
Jul 30 17:08:11.271299 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:08:11.271271    2555 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 30 17:08:11.271424 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:08:11.271332    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79f9bec6-db1a-4830-b663-2f3435132f67-cert podName:79f9bec6-db1a-4830-b663-2f3435132f67 nodeName:}" failed. No retries permitted until 2026-07-30 17:08:11.771311015 +0000 UTC m=+466.258678395 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/79f9bec6-db1a-4830-b663-2f3435132f67-cert") pod "odh-model-controller-8588677cfb-zbjs8" (UID: "79f9bec6-db1a-4830-b663-2f3435132f67") : secret "odh-model-controller-webhook-cert" not found
Jul 30 17:08:11.272905 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:11.272880    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:08:11.273548 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:11.273520    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79f9bec6-db1a-4830-b663-2f3435132f67-metrics-certs\") pod \"odh-model-controller-8588677cfb-zbjs8\" (UID: \"79f9bec6-db1a-4830-b663-2f3435132f67\") " pod="kserve/odh-model-controller-8588677cfb-zbjs8"
Jul 30 17:08:11.273647 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:11.273594    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d30e976-fa6b-44af-9a2a-5d8433a3cd84-tls-certs\") pod \"model-serving-api-7b954ffc95-kvr8r\" (UID: \"0d30e976-fa6b-44af-9a2a-5d8433a3cd84\") " pod="kserve/model-serving-api-7b954ffc95-kvr8r"
Jul 30 17:08:11.282114 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:11.282092    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj29b\" (UniqueName: \"kubernetes.io/projected/0d30e976-fa6b-44af-9a2a-5d8433a3cd84-kube-api-access-pj29b\") pod \"model-serving-api-7b954ffc95-kvr8r\" (UID: \"0d30e976-fa6b-44af-9a2a-5d8433a3cd84\") " pod="kserve/model-serving-api-7b954ffc95-kvr8r"
Jul 30 17:08:11.283067 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:11.282996    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw8jz\" (UniqueName: \"kubernetes.io/projected/79f9bec6-db1a-4830-b663-2f3435132f67-kube-api-access-hw8jz\") pod \"odh-model-controller-8588677cfb-zbjs8\" (UID: \"79f9bec6-db1a-4830-b663-2f3435132f67\") " pod="kserve/odh-model-controller-8588677cfb-zbjs8"
Jul 30 17:08:11.369480 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:11.369446    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:08:11.395893 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:11.395870    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-kvr8r"
Jul 30 17:08:11.525094 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:11.525072    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-kvr8r"]
Jul 30 17:08:11.527343 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:08:11.527314    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d30e976_fa6b_44af_9a2a_5d8433a3cd84.slice/crio-0f46748ad744888e57cd0aa970f4c3ab0721ff19dd829eb99aa90851d7e2eace WatchSource:0}: Error finding container 0f46748ad744888e57cd0aa970f4c3ab0721ff19dd829eb99aa90851d7e2eace: Status 404 returned error can't find the container with id 0f46748ad744888e57cd0aa970f4c3ab0721ff19dd829eb99aa90851d7e2eace
Jul 30 17:08:11.673945 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:11.673871    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:08:11.773858 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:11.773827    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:08:11.775497 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:11.775479    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/79f9bec6-db1a-4830-b663-2f3435132f67-cert\") pod \"odh-model-controller-8588677cfb-zbjs8\" (UID: \"79f9bec6-db1a-4830-b663-2f3435132f67\") " pod="kserve/odh-model-controller-8588677cfb-zbjs8"
Jul 30 17:08:11.777764 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:11.777747    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/79f9bec6-db1a-4830-b663-2f3435132f67-cert\") pod \"odh-model-controller-8588677cfb-zbjs8\" (UID: \"79f9bec6-db1a-4830-b663-2f3435132f67\") " pod="kserve/odh-model-controller-8588677cfb-zbjs8"
Jul 30 17:08:12.011715 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:12.011630    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-8588677cfb-zbjs8"
Jul 30 17:08:12.150548 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:12.150442    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-8588677cfb-zbjs8"]
Jul 30 17:08:12.154088 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:08:12.154053    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79f9bec6_db1a_4830_b663_2f3435132f67.slice/crio-b09329002e4387c272b117c105a171560305ac11abe3d829a21a6e101576e73b WatchSource:0}: Error finding container b09329002e4387c272b117c105a171560305ac11abe3d829a21a6e101576e73b: Status 404 returned error can't find the container with id b09329002e4387c272b117c105a171560305ac11abe3d829a21a6e101576e73b
Jul 30 17:08:12.154590 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:12.154560    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-kvr8r" event={"ID":"0d30e976-fa6b-44af-9a2a-5d8433a3cd84","Type":"ContainerStarted","Data":"0f46748ad744888e57cd0aa970f4c3ab0721ff19dd829eb99aa90851d7e2eace"}
Jul 30 17:08:13.162220 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:13.162039    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-8588677cfb-zbjs8" event={"ID":"79f9bec6-db1a-4830-b663-2f3435132f67","Type":"ContainerStarted","Data":"b09329002e4387c272b117c105a171560305ac11abe3d829a21a6e101576e73b"}
Jul 30 17:08:13.163902 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:13.163842    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-kvr8r" event={"ID":"0d30e976-fa6b-44af-9a2a-5d8433a3cd84","Type":"ContainerStarted","Data":"891f63f293ee13fabf9d425afe274edae74783826d4536e7b0c34ce00b088192"}
Jul 30 17:08:13.164066 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:13.163971    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-kvr8r"
Jul 30 17:08:13.186108 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:13.185960    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-kvr8r" podStartSLOduration=0.795597171 podStartE2EDuration="2.185944281s" totalImagesPullingTime="1.39034711s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:08:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:08:13.182889096 +0000 UTC m=+467.670256520" watchObservedRunningTime="2026-07-30 17:08:13.185944281 +0000 UTC m=+467.673311706"
Jul 30 17:08:15.161501 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:15.161467    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-857dfd46db-84ghf"
Jul 30 17:08:15.161877 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:15.161528    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-857dfd46db-84ghf"
Jul 30 17:08:15.166035 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:15.166000    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-857dfd46db-84ghf"
Jul 30 17:08:15.175647 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:15.175621    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-8588677cfb-zbjs8" event={"ID":"79f9bec6-db1a-4830-b663-2f3435132f67","Type":"ContainerStarted","Data":"a10f4b7aa12546e0b9641564b1e2b8b6501ae8b5f5b5b5be445e32ce44c05228"}
Jul 30 17:08:15.175770 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:15.175757    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-8588677cfb-zbjs8"
Jul 30 17:08:15.179516 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:15.179495    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-857dfd46db-84ghf"
Jul 30 17:08:15.198668 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:15.198623    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-8588677cfb-zbjs8" podStartSLOduration=1.315354392 podStartE2EDuration="4.198606117s" totalImagesPullingTime="2.883251725s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:08:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:08:15.197865323 +0000 UTC m=+469.685232726" watchObservedRunningTime="2026-07-30 17:08:15.198606117 +0000 UTC m=+469.685973518"
Jul 30 17:08:15.242445 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:15.242404    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68f68885b6-n7n7m"]
Jul 30 17:08:15.344283 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:15.344244    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:08:24.173945 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:24.173913    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-kvr8r"
Jul 30 17:08:26.183116 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:26.183085    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-8588677cfb-zbjs8"
Jul 30 17:08:40.261360 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:40.261302    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-68f68885b6-n7n7m" podUID="3f441ce7-8a25-41b8-a8c1-28fb65d36f0f" containerName="console" containerID="cri-o://75a7d95e94f57391b46552fcea8786da569fff1ecb5e8039d0f1b52f9c61ee29" gracePeriod=15
Jul 30 17:08:40.503619 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:40.503595    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68f68885b6-n7n7m"
Jul 30 17:08:40.632628 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:40.632593    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3f441ce7-8a25-41b8-a8c1-28fb65d36f0f-console-config\") pod \"3f441ce7-8a25-41b8-a8c1-28fb65d36f0f\" (UID: \"3f441ce7-8a25-41b8-a8c1-28fb65d36f0f\") "
Jul 30 17:08:40.632628 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:40.632631    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3f441ce7-8a25-41b8-a8c1-28fb65d36f0f-console-oauth-config\") pod \"3f441ce7-8a25-41b8-a8c1-28fb65d36f0f\" (UID: \"3f441ce7-8a25-41b8-a8c1-28fb65d36f0f\") "
Jul 30 17:08:40.632892 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:40.632651    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9cgfh\" (UniqueName: \"kubernetes.io/projected/3f441ce7-8a25-41b8-a8c1-28fb65d36f0f-kube-api-access-9cgfh\") pod \"3f441ce7-8a25-41b8-a8c1-28fb65d36f0f\" (UID: \"3f441ce7-8a25-41b8-a8c1-28fb65d36f0f\") "
Jul 30 17:08:40.632892 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:40.632673    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3f441ce7-8a25-41b8-a8c1-28fb65d36f0f-service-ca\") pod \"3f441ce7-8a25-41b8-a8c1-28fb65d36f0f\" (UID: \"3f441ce7-8a25-41b8-a8c1-28fb65d36f0f\") "
Jul 30 17:08:40.632892 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:40.632696    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3f441ce7-8a25-41b8-a8c1-28fb65d36f0f-console-serving-cert\") pod \"3f441ce7-8a25-41b8-a8c1-28fb65d36f0f\" (UID: \"3f441ce7-8a25-41b8-a8c1-28fb65d36f0f\") "
Jul 30 17:08:40.632892 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:40.632735    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3f441ce7-8a25-41b8-a8c1-28fb65d36f0f-oauth-serving-cert\") pod \"3f441ce7-8a25-41b8-a8c1-28fb65d36f0f\" (UID: \"3f441ce7-8a25-41b8-a8c1-28fb65d36f0f\") "
Jul 30 17:08:40.632892 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:40.632750    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f441ce7-8a25-41b8-a8c1-28fb65d36f0f-trusted-ca-bundle\") pod \"3f441ce7-8a25-41b8-a8c1-28fb65d36f0f\" (UID: \"3f441ce7-8a25-41b8-a8c1-28fb65d36f0f\") "
Jul 30 17:08:40.633191 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:40.633059    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f441ce7-8a25-41b8-a8c1-28fb65d36f0f-console-config" (OuterVolumeSpecName: "console-config") pod "3f441ce7-8a25-41b8-a8c1-28fb65d36f0f" (UID: "3f441ce7-8a25-41b8-a8c1-28fb65d36f0f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:08:40.633249 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:40.633201    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f441ce7-8a25-41b8-a8c1-28fb65d36f0f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "3f441ce7-8a25-41b8-a8c1-28fb65d36f0f" (UID: "3f441ce7-8a25-41b8-a8c1-28fb65d36f0f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:08:40.633299 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:40.633238    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f441ce7-8a25-41b8-a8c1-28fb65d36f0f-service-ca" (OuterVolumeSpecName: "service-ca") pod "3f441ce7-8a25-41b8-a8c1-28fb65d36f0f" (UID: "3f441ce7-8a25-41b8-a8c1-28fb65d36f0f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:08:40.633348 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:40.633311    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f441ce7-8a25-41b8-a8c1-28fb65d36f0f-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "3f441ce7-8a25-41b8-a8c1-28fb65d36f0f" (UID: "3f441ce7-8a25-41b8-a8c1-28fb65d36f0f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:08:40.634892 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:40.634863    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f441ce7-8a25-41b8-a8c1-28fb65d36f0f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "3f441ce7-8a25-41b8-a8c1-28fb65d36f0f" (UID: "3f441ce7-8a25-41b8-a8c1-28fb65d36f0f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:08:40.634892 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:40.634880    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f441ce7-8a25-41b8-a8c1-28fb65d36f0f-kube-api-access-9cgfh" (OuterVolumeSpecName: "kube-api-access-9cgfh") pod "3f441ce7-8a25-41b8-a8c1-28fb65d36f0f" (UID: "3f441ce7-8a25-41b8-a8c1-28fb65d36f0f"). InnerVolumeSpecName "kube-api-access-9cgfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 17:08:40.635078 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:40.634890    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f441ce7-8a25-41b8-a8c1-28fb65d36f0f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "3f441ce7-8a25-41b8-a8c1-28fb65d36f0f" (UID: "3f441ce7-8a25-41b8-a8c1-28fb65d36f0f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:08:40.733475 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:40.733440    2555 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3f441ce7-8a25-41b8-a8c1-28fb65d36f0f-console-config\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:08:40.733475 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:40.733469    2555 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3f441ce7-8a25-41b8-a8c1-28fb65d36f0f-console-oauth-config\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:08:40.733667 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:40.733483    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cgfh\" (UniqueName: \"kubernetes.io/projected/3f441ce7-8a25-41b8-a8c1-28fb65d36f0f-kube-api-access-9cgfh\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:08:40.733667 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:40.733497    2555 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3f441ce7-8a25-41b8-a8c1-28fb65d36f0f-service-ca\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:08:40.733667 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:40.733509    2555 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3f441ce7-8a25-41b8-a8c1-28fb65d36f0f-console-serving-cert\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:08:40.733667 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:40.733520    2555 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3f441ce7-8a25-41b8-a8c1-28fb65d36f0f-oauth-serving-cert\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:08:40.733667 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:40.733532    2555 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f441ce7-8a25-41b8-a8c1-28fb65d36f0f-trusted-ca-bundle\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:08:41.293319 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:41.293287    2555 generic.go:356] "Generic (PLEG): container finished" podID="3f441ce7-8a25-41b8-a8c1-28fb65d36f0f" containerID="75a7d95e94f57391b46552fcea8786da569fff1ecb5e8039d0f1b52f9c61ee29" exitCode=0
Jul 30 17:08:41.293769 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:41.293336    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68f68885b6-n7n7m"
Jul 30 17:08:41.293769 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:41.293364    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68f68885b6-n7n7m" event={"ID":"3f441ce7-8a25-41b8-a8c1-28fb65d36f0f","Type":"ContainerDied","Data":"75a7d95e94f57391b46552fcea8786da569fff1ecb5e8039d0f1b52f9c61ee29"}
Jul 30 17:08:41.293769 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:41.293414    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68f68885b6-n7n7m" event={"ID":"3f441ce7-8a25-41b8-a8c1-28fb65d36f0f","Type":"ContainerDied","Data":"cee505941ce06740922c5e679923433d6f299aeed2bb8597f8b3dd96402a68dc"}
Jul 30 17:08:41.293769 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:41.293434    2555 scope.go:117] "RemoveContainer" containerID="75a7d95e94f57391b46552fcea8786da569fff1ecb5e8039d0f1b52f9c61ee29"
Jul 30 17:08:41.302615 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:41.302600    2555 scope.go:117] "RemoveContainer" containerID="75a7d95e94f57391b46552fcea8786da569fff1ecb5e8039d0f1b52f9c61ee29"
Jul 30 17:08:41.302883 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:08:41.302864    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75a7d95e94f57391b46552fcea8786da569fff1ecb5e8039d0f1b52f9c61ee29\": container with ID starting with 75a7d95e94f57391b46552fcea8786da569fff1ecb5e8039d0f1b52f9c61ee29 not found: ID does not exist" containerID="75a7d95e94f57391b46552fcea8786da569fff1ecb5e8039d0f1b52f9c61ee29"
Jul 30 17:08:41.302950 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:41.302890    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75a7d95e94f57391b46552fcea8786da569fff1ecb5e8039d0f1b52f9c61ee29"} err="failed to get container status \"75a7d95e94f57391b46552fcea8786da569fff1ecb5e8039d0f1b52f9c61ee29\": rpc error: code = NotFound desc = could not find container \"75a7d95e94f57391b46552fcea8786da569fff1ecb5e8039d0f1b52f9c61ee29\": container with ID starting with 75a7d95e94f57391b46552fcea8786da569fff1ecb5e8039d0f1b52f9c61ee29 not found: ID does not exist"
Jul 30 17:08:41.314823 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:41.314801    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68f68885b6-n7n7m"]
Jul 30 17:08:41.316844 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:41.316826    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-68f68885b6-n7n7m"]
Jul 30 17:08:42.060471 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:08:42.060441    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f441ce7-8a25-41b8-a8c1-28fb65d36f0f" path="/var/lib/kubelet/pods/3f441ce7-8a25-41b8-a8c1-28fb65d36f0f/volumes"
Jul 30 17:09:40.900269 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:09:40.900236    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:09:46.473023 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:09:46.472982    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-n7hf2"]
Jul 30 17:09:46.473702 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:09:46.473383    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f441ce7-8a25-41b8-a8c1-28fb65d36f0f" containerName="console"
Jul 30 17:09:46.473702 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:09:46.473395    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f441ce7-8a25-41b8-a8c1-28fb65d36f0f" containerName="console"
Jul 30 17:09:46.473702 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:09:46.473468    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f441ce7-8a25-41b8-a8c1-28fb65d36f0f" containerName="console"
Jul 30 17:09:46.477428 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:09:46.477413    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-n7hf2"
Jul 30 17:09:46.479408 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:09:46.479386    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:09:46.479636 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:09:46.479621    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Jul 30 17:09:46.483469 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:09:46.483440    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-n7hf2"]
Jul 30 17:09:46.598040 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:09:46.597988    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c495s\" (UniqueName: \"kubernetes.io/projected/38dbb77e-99ba-4583-87cd-c8f5f389b767-kube-api-access-c495s\") pod \"seaweedfs-tls-custom-6644bf5d6c-n7hf2\" (UID: \"38dbb77e-99ba-4583-87cd-c8f5f389b767\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-n7hf2"
Jul 30 17:09:46.598239 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:09:46.598059    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/38dbb77e-99ba-4583-87cd-c8f5f389b767-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-n7hf2\" (UID: \"38dbb77e-99ba-4583-87cd-c8f5f389b767\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-n7hf2"
Jul 30 17:09:46.699378 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:09:46.699341    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c495s\" (UniqueName: \"kubernetes.io/projected/38dbb77e-99ba-4583-87cd-c8f5f389b767-kube-api-access-c495s\") pod \"seaweedfs-tls-custom-6644bf5d6c-n7hf2\" (UID: \"38dbb77e-99ba-4583-87cd-c8f5f389b767\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-n7hf2"
Jul 30 17:09:46.699378 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:09:46.699379    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/38dbb77e-99ba-4583-87cd-c8f5f389b767-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-n7hf2\" (UID: \"38dbb77e-99ba-4583-87cd-c8f5f389b767\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-n7hf2"
Jul 30 17:09:46.699692 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:09:46.699678    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/38dbb77e-99ba-4583-87cd-c8f5f389b767-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-n7hf2\" (UID: \"38dbb77e-99ba-4583-87cd-c8f5f389b767\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-n7hf2"
Jul 30 17:09:46.709836 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:09:46.709815    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c495s\" (UniqueName: \"kubernetes.io/projected/38dbb77e-99ba-4583-87cd-c8f5f389b767-kube-api-access-c495s\") pod \"seaweedfs-tls-custom-6644bf5d6c-n7hf2\" (UID: \"38dbb77e-99ba-4583-87cd-c8f5f389b767\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-n7hf2"
Jul 30 17:09:46.786840 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:09:46.786767    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-n7hf2"
Jul 30 17:09:46.917257 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:09:46.917233    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-n7hf2"]
Jul 30 17:09:46.919814 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:09:46.919789    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38dbb77e_99ba_4583_87cd_c8f5f389b767.slice/crio-e9b64e9c2fe2a1d81ddd2fb7d01fd9732206dd5b75ae5cf0df44d27f954ee940 WatchSource:0}: Error finding container e9b64e9c2fe2a1d81ddd2fb7d01fd9732206dd5b75ae5cf0df44d27f954ee940: Status 404 returned error can't find the container with id e9b64e9c2fe2a1d81ddd2fb7d01fd9732206dd5b75ae5cf0df44d27f954ee940
Jul 30 17:09:47.584652 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:09:47.584612    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-n7hf2" event={"ID":"38dbb77e-99ba-4583-87cd-c8f5f389b767","Type":"ContainerStarted","Data":"e6d920b1733ccfadff1b2a23927b4e0cd97ec94bb5df7f339c2082fbcc63aef3"}
Jul 30 17:09:47.584652 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:09:47.584661    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-n7hf2" event={"ID":"38dbb77e-99ba-4583-87cd-c8f5f389b767","Type":"ContainerStarted","Data":"e9b64e9c2fe2a1d81ddd2fb7d01fd9732206dd5b75ae5cf0df44d27f954ee940"}
Jul 30 17:09:47.602759 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:09:47.602713    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-n7hf2" podStartSLOduration=1.2665125640000001 podStartE2EDuration="1.602697602s" totalImagesPullingTime="336.185038ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:09:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:09:47.600520134 +0000 UTC m=+562.087887536" watchObservedRunningTime="2026-07-30 17:09:47.602697602 +0000 UTC m=+562.090065002"
Jul 30 17:09:49.206644 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:09:49.206609    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-n7hf2"]
Jul 30 17:09:49.593070 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:09:49.593034    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-n7hf2" podUID="38dbb77e-99ba-4583-87cd-c8f5f389b767" containerName="seaweedfs-tls-custom" containerID="cri-o://e6d920b1733ccfadff1b2a23927b4e0cd97ec94bb5df7f339c2082fbcc63aef3" gracePeriod=30
Jul 30 17:09:50.837187 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:09:50.837162    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-n7hf2"
Jul 30 17:09:50.932929 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:09:50.932846    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/38dbb77e-99ba-4583-87cd-c8f5f389b767-data\") pod \"38dbb77e-99ba-4583-87cd-c8f5f389b767\" (UID: \"38dbb77e-99ba-4583-87cd-c8f5f389b767\") "
Jul 30 17:09:50.932929 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:09:50.932900    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c495s\" (UniqueName: \"kubernetes.io/projected/38dbb77e-99ba-4583-87cd-c8f5f389b767-kube-api-access-c495s\") pod \"38dbb77e-99ba-4583-87cd-c8f5f389b767\" (UID: \"38dbb77e-99ba-4583-87cd-c8f5f389b767\") "
Jul 30 17:09:50.934115 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:09:50.934092    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38dbb77e-99ba-4583-87cd-c8f5f389b767-data" (OuterVolumeSpecName: "data") pod "38dbb77e-99ba-4583-87cd-c8f5f389b767" (UID: "38dbb77e-99ba-4583-87cd-c8f5f389b767"). InnerVolumeSpecName "data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 17:09:50.934846 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:09:50.934816    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38dbb77e-99ba-4583-87cd-c8f5f389b767-kube-api-access-c495s" (OuterVolumeSpecName: "kube-api-access-c495s") pod "38dbb77e-99ba-4583-87cd-c8f5f389b767" (UID: "38dbb77e-99ba-4583-87cd-c8f5f389b767"). InnerVolumeSpecName "kube-api-access-c495s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 17:09:51.034444 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:09:51.034399    2555 reconciler_common.go:299] "Volume detached for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/38dbb77e-99ba-4583-87cd-c8f5f389b767-data\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:09:51.034444 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:09:51.034439    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c495s\" (UniqueName: \"kubernetes.io/projected/38dbb77e-99ba-4583-87cd-c8f5f389b767-kube-api-access-c495s\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:09:51.603119 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:09:51.603085    2555 generic.go:356] "Generic (PLEG): container finished" podID="38dbb77e-99ba-4583-87cd-c8f5f389b767" containerID="e6d920b1733ccfadff1b2a23927b4e0cd97ec94bb5df7f339c2082fbcc63aef3" exitCode=0
Jul 30 17:09:51.603332 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:09:51.603130    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-n7hf2"
Jul 30 17:09:51.603332 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:09:51.603159    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-n7hf2" event={"ID":"38dbb77e-99ba-4583-87cd-c8f5f389b767","Type":"ContainerDied","Data":"e6d920b1733ccfadff1b2a23927b4e0cd97ec94bb5df7f339c2082fbcc63aef3"}
Jul 30 17:09:51.603332 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:09:51.603208    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-n7hf2" event={"ID":"38dbb77e-99ba-4583-87cd-c8f5f389b767","Type":"ContainerDied","Data":"e9b64e9c2fe2a1d81ddd2fb7d01fd9732206dd5b75ae5cf0df44d27f954ee940"}
Jul 30 17:09:51.603332 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:09:51.603234    2555 scope.go:117] "RemoveContainer" containerID="e6d920b1733ccfadff1b2a23927b4e0cd97ec94bb5df7f339c2082fbcc63aef3"
Jul 30 17:09:51.612663 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:09:51.612645    2555 scope.go:117] "RemoveContainer" containerID="e6d920b1733ccfadff1b2a23927b4e0cd97ec94bb5df7f339c2082fbcc63aef3"
Jul 30 17:09:51.612903 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:09:51.612884    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6d920b1733ccfadff1b2a23927b4e0cd97ec94bb5df7f339c2082fbcc63aef3\": container with ID starting with e6d920b1733ccfadff1b2a23927b4e0cd97ec94bb5df7f339c2082fbcc63aef3 not found: ID does not exist" containerID="e6d920b1733ccfadff1b2a23927b4e0cd97ec94bb5df7f339c2082fbcc63aef3"
Jul 30 17:09:51.612951 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:09:51.612912    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6d920b1733ccfadff1b2a23927b4e0cd97ec94bb5df7f339c2082fbcc63aef3"} err="failed to get container status \"e6d920b1733ccfadff1b2a23927b4e0cd97ec94bb5df7f339c2082fbcc63aef3\": rpc error: code = NotFound desc = could not find container \"e6d920b1733ccfadff1b2a23927b4e0cd97ec94bb5df7f339c2082fbcc63aef3\": container with ID starting with e6d920b1733ccfadff1b2a23927b4e0cd97ec94bb5df7f339c2082fbcc63aef3 not found: ID does not exist"
Jul 30 17:09:51.624818 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:09:51.624792    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-n7hf2"]
Jul 30 17:09:51.632941 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:09:51.632919    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-n7hf2"]
Jul 30 17:09:51.658918 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:09:51.658894    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-rwg8w"]
Jul 30 17:09:51.659264 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:09:51.659250    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38dbb77e-99ba-4583-87cd-c8f5f389b767" containerName="seaweedfs-tls-custom"
Jul 30 17:09:51.659264 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:09:51.659264    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="38dbb77e-99ba-4583-87cd-c8f5f389b767" containerName="seaweedfs-tls-custom"
Jul 30 17:09:51.659363 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:09:51.659336    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="38dbb77e-99ba-4583-87cd-c8f5f389b767" containerName="seaweedfs-tls-custom"
Jul 30 17:09:51.663813 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:09:51.663797    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-rwg8w"
Jul 30 17:09:51.666468 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:09:51.666453    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom\""
Jul 30 17:09:51.666530 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:09:51.666462    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Jul 30 17:09:51.669432 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:09:51.669412    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-rwg8w"]
Jul 30 17:09:51.742085 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:09:51.742043    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/1a9ee89d-0575-405a-8d22-b0e4070283f6-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-rwg8w\" (UID: \"1a9ee89d-0575-405a-8d22-b0e4070283f6\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-rwg8w"
Jul 30 17:09:51.742085 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:09:51.742104    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq5s8\" (UniqueName: \"kubernetes.io/projected/1a9ee89d-0575-405a-8d22-b0e4070283f6-kube-api-access-xq5s8\") pod \"seaweedfs-tls-custom-8b44d767d-rwg8w\" (UID: \"1a9ee89d-0575-405a-8d22-b0e4070283f6\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-rwg8w"
Jul 30 17:09:51.742316 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:09:51.742175    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1a9ee89d-0575-405a-8d22-b0e4070283f6-data\") pod \"seaweedfs-tls-custom-8b44d767d-rwg8w\" (UID: \"1a9ee89d-0575-405a-8d22-b0e4070283f6\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-rwg8w"
Jul 30 17:09:51.842594 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:09:51.842564    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/1a9ee89d-0575-405a-8d22-b0e4070283f6-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-rwg8w\" (UID: \"1a9ee89d-0575-405a-8d22-b0e4070283f6\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-rwg8w"
Jul 30 17:09:51.842994 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:09:51.842613    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq5s8\" (UniqueName: \"kubernetes.io/projected/1a9ee89d-0575-405a-8d22-b0e4070283f6-kube-api-access-xq5s8\") pod \"seaweedfs-tls-custom-8b44d767d-rwg8w\" (UID: \"1a9ee89d-0575-405a-8d22-b0e4070283f6\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-rwg8w"
Jul 30 17:09:51.842994 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:09:51.842645    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1a9ee89d-0575-405a-8d22-b0e4070283f6-data\") pod \"seaweedfs-tls-custom-8b44d767d-rwg8w\" (UID: \"1a9ee89d-0575-405a-8d22-b0e4070283f6\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-rwg8w"
Jul 30 17:09:51.842994 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:09:51.842969    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1a9ee89d-0575-405a-8d22-b0e4070283f6-data\") pod \"seaweedfs-tls-custom-8b44d767d-rwg8w\" (UID: \"1a9ee89d-0575-405a-8d22-b0e4070283f6\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-rwg8w"
Jul 30 17:09:51.844981 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:09:51.844963    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/1a9ee89d-0575-405a-8d22-b0e4070283f6-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-rwg8w\" (UID: \"1a9ee89d-0575-405a-8d22-b0e4070283f6\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-rwg8w"
Jul 30 17:09:51.852768 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:09:51.852745    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq5s8\" (UniqueName: \"kubernetes.io/projected/1a9ee89d-0575-405a-8d22-b0e4070283f6-kube-api-access-xq5s8\") pod \"seaweedfs-tls-custom-8b44d767d-rwg8w\" (UID: \"1a9ee89d-0575-405a-8d22-b0e4070283f6\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-rwg8w"
Jul 30 17:09:51.973575 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:09:51.973492    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-rwg8w"
Jul 30 17:09:52.062027 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:09:52.061979    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38dbb77e-99ba-4583-87cd-c8f5f389b767" path="/var/lib/kubelet/pods/38dbb77e-99ba-4583-87cd-c8f5f389b767/volumes"
Jul 30 17:09:52.104753 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:09:52.104707    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-rwg8w"]
Jul 30 17:09:52.106814 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:09:52.106792    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a9ee89d_0575_405a_8d22_b0e4070283f6.slice/crio-03349aa5a74bc9e5957313324c1c9dcfa5fca057ee5e8559bf502cbb7a65b94f WatchSource:0}: Error finding container 03349aa5a74bc9e5957313324c1c9dcfa5fca057ee5e8559bf502cbb7a65b94f: Status 404 returned error can't find the container with id 03349aa5a74bc9e5957313324c1c9dcfa5fca057ee5e8559bf502cbb7a65b94f
Jul 30 17:09:52.610126 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:09:52.610092    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-rwg8w" event={"ID":"1a9ee89d-0575-405a-8d22-b0e4070283f6","Type":"ContainerStarted","Data":"4f5b6d475c3da35b199b0b17584847ac727649290da9c06dedcd4f90fa2da44f"}
Jul 30 17:09:52.610311 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:09:52.610143    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-rwg8w" event={"ID":"1a9ee89d-0575-405a-8d22-b0e4070283f6","Type":"ContainerStarted","Data":"03349aa5a74bc9e5957313324c1c9dcfa5fca057ee5e8559bf502cbb7a65b94f"}
Jul 30 17:09:52.627541 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:09:52.627492    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-8b44d767d-rwg8w" podStartSLOduration=1.352198686 podStartE2EDuration="1.627476427s" totalImagesPullingTime="275.277741ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:09:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:09:52.626295128 +0000 UTC m=+567.113662529" watchObservedRunningTime="2026-07-30 17:09:52.627476427 +0000 UTC m=+567.114843827"
Jul 30 17:09:59.952320 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:09:59.952289    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:10:04.447369 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:10:04.447340    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-n52p8"]
Jul 30 17:10:04.450811 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:10:04.450795    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-n52p8"
Jul 30 17:10:04.453184 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:10:04.453163    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving\""
Jul 30 17:10:04.453307 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:10:04.453224    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Jul 30 17:10:04.464384 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:10:04.464356    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:10:04.464967 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:10:04.464946    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-n52p8"]
Jul 30 17:10:04.556329 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:10:04.556294    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/af454a5f-6e99-4da4-98a8-ea7cde369acc-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-n52p8\" (UID: \"af454a5f-6e99-4da4-98a8-ea7cde369acc\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-n52p8"
Jul 30 17:10:04.556500 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:10:04.556370    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpbzf\" (UniqueName: \"kubernetes.io/projected/af454a5f-6e99-4da4-98a8-ea7cde369acc-kube-api-access-hpbzf\") pod \"seaweedfs-tls-serving-67b7477c6d-n52p8\" (UID: \"af454a5f-6e99-4da4-98a8-ea7cde369acc\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-n52p8"
Jul 30 17:10:04.556500 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:10:04.556415    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/af454a5f-6e99-4da4-98a8-ea7cde369acc-data\") pod \"seaweedfs-tls-serving-67b7477c6d-n52p8\" (UID: \"af454a5f-6e99-4da4-98a8-ea7cde369acc\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-n52p8"
Jul 30 17:10:04.657044 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:10:04.656991    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpbzf\" (UniqueName: \"kubernetes.io/projected/af454a5f-6e99-4da4-98a8-ea7cde369acc-kube-api-access-hpbzf\") pod \"seaweedfs-tls-serving-67b7477c6d-n52p8\" (UID: \"af454a5f-6e99-4da4-98a8-ea7cde369acc\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-n52p8"
Jul 30 17:10:04.657197 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:10:04.657055    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/af454a5f-6e99-4da4-98a8-ea7cde369acc-data\") pod \"seaweedfs-tls-serving-67b7477c6d-n52p8\" (UID: \"af454a5f-6e99-4da4-98a8-ea7cde369acc\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-n52p8"
Jul 30 17:10:04.657197 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:10:04.657106    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/af454a5f-6e99-4da4-98a8-ea7cde369acc-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-n52p8\" (UID: \"af454a5f-6e99-4da4-98a8-ea7cde369acc\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-n52p8"
Jul 30 17:10:04.657272 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:10:04.657213    2555 projected.go:266] Couldn't get secret kserve/seaweedfs-tls-serving: secret "seaweedfs-tls-serving" not found
Jul 30 17:10:04.657272 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:10:04.657224    2555 projected.go:196] Error preparing data for projected volume seaweedfs-tls-serving for pod kserve/seaweedfs-tls-serving-67b7477c6d-n52p8: secret "seaweedfs-tls-serving" not found
Jul 30 17:10:04.657334 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:10:04.657279    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af454a5f-6e99-4da4-98a8-ea7cde369acc-seaweedfs-tls-serving podName:af454a5f-6e99-4da4-98a8-ea7cde369acc nodeName:}" failed. No retries permitted until 2026-07-30 17:10:05.15726431 +0000 UTC m=+579.644631690 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "seaweedfs-tls-serving" (UniqueName: "kubernetes.io/projected/af454a5f-6e99-4da4-98a8-ea7cde369acc-seaweedfs-tls-serving") pod "seaweedfs-tls-serving-67b7477c6d-n52p8" (UID: "af454a5f-6e99-4da4-98a8-ea7cde369acc") : secret "seaweedfs-tls-serving" not found
Jul 30 17:10:04.657375 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:10:04.657357    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/af454a5f-6e99-4da4-98a8-ea7cde369acc-data\") pod \"seaweedfs-tls-serving-67b7477c6d-n52p8\" (UID: \"af454a5f-6e99-4da4-98a8-ea7cde369acc\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-n52p8"
Jul 30 17:10:04.666467 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:10:04.666438    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpbzf\" (UniqueName: \"kubernetes.io/projected/af454a5f-6e99-4da4-98a8-ea7cde369acc-kube-api-access-hpbzf\") pod \"seaweedfs-tls-serving-67b7477c6d-n52p8\" (UID: \"af454a5f-6e99-4da4-98a8-ea7cde369acc\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-n52p8"
Jul 30 17:10:04.904483 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:10:04.904454    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:10:05.160382 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:10:05.160297    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/af454a5f-6e99-4da4-98a8-ea7cde369acc-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-n52p8\" (UID: \"af454a5f-6e99-4da4-98a8-ea7cde369acc\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-n52p8"
Jul 30 17:10:05.162620 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:10:05.162588    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/af454a5f-6e99-4da4-98a8-ea7cde369acc-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-n52p8\" (UID: \"af454a5f-6e99-4da4-98a8-ea7cde369acc\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-n52p8"
Jul 30 17:10:05.360478 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:10:05.360444    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-n52p8"
Jul 30 17:10:05.487515 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:10:05.487489    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-n52p8"]
Jul 30 17:10:05.489686 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:10:05.489661    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf454a5f_6e99_4da4_98a8_ea7cde369acc.slice/crio-9e83fad3033b47e40892b449e793d6a5d2dbb14eea7a714f2a3feef5f19e3fe2 WatchSource:0}: Error finding container 9e83fad3033b47e40892b449e793d6a5d2dbb14eea7a714f2a3feef5f19e3fe2: Status 404 returned error can't find the container with id 9e83fad3033b47e40892b449e793d6a5d2dbb14eea7a714f2a3feef5f19e3fe2
Jul 30 17:10:05.672171 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:10:05.672135    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-n52p8" event={"ID":"af454a5f-6e99-4da4-98a8-ea7cde369acc","Type":"ContainerStarted","Data":"9e83fad3033b47e40892b449e793d6a5d2dbb14eea7a714f2a3feef5f19e3fe2"}
Jul 30 17:10:06.678320 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:10:06.678288    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-n52p8" event={"ID":"af454a5f-6e99-4da4-98a8-ea7cde369acc","Type":"ContainerStarted","Data":"04e29715c1013358df97e50772013deea402205f2fa1bbe558e6c51569048f13"}
Jul 30 17:10:06.696548 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:10:06.696504    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-serving-67b7477c6d-n52p8" podStartSLOduration=2.437252508 podStartE2EDuration="2.696488025s" totalImagesPullingTime="259.235517ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:10:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:10:06.695250847 +0000 UTC m=+581.182618247" watchObservedRunningTime="2026-07-30 17:10:06.696488025 +0000 UTC m=+581.183855425"
Jul 30 17:10:15.709061 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:10:15.709024    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:10:20.229864 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:10:20.229831    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:10:20.327065 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:10:20.327036    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:10:26.224111 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:10:26.224075    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:10:31.508715 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:10:31.508682    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:10:31.889556 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:10:31.889529    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:10:35.478150 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:10:35.478112    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:11:22.518162 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:11:22.518124    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:11:22.569350 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:11:22.569317    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:12:16.667746 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:12:16.667673    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:12:16.750330 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:12:16.750300    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:12:16.957283 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:12:16.957208    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:13:51.884239 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:13:51.884158    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:13:51.899278 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:13:51.899249    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-8c5sj"]
Jul 30 17:13:51.902936 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:13:51.902916    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-8c5sj"
Jul 30 17:13:51.905594 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:13:51.905572    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 30 17:13:51.905723 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:13:51.905613    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-predictor-serving-cert\""
Jul 30 17:13:51.906972 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:13:51.906953    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-kube-rbac-proxy-sar-config\""
Jul 30 17:13:51.907109 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:13:51.906978    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 30 17:13:51.907109 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:13:51.906953    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-plw5t\""
Jul 30 17:13:51.915114 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:13:51.915092    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-8c5sj"]
Jul 30 17:13:51.924499 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:13:51.924476    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a981a8dd-5655-4d2d-9a6f-2ffa0eca0496-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-8c5sj\" (UID: \"a981a8dd-5655-4d2d-9a6f-2ffa0eca0496\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-8c5sj"
Jul 30 17:13:51.924601 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:13:51.924539    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a981a8dd-5655-4d2d-9a6f-2ffa0eca0496-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-8c5sj\" (UID: \"a981a8dd-5655-4d2d-9a6f-2ffa0eca0496\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-8c5sj"
Jul 30 17:13:51.924601 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:13:51.924569    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5czn\" (UniqueName: \"kubernetes.io/projected/a981a8dd-5655-4d2d-9a6f-2ffa0eca0496-kube-api-access-j5czn\") pod \"message-dumper-predictor-6575bcfdfd-8c5sj\" (UID: \"a981a8dd-5655-4d2d-9a6f-2ffa0eca0496\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-8c5sj"
Jul 30 17:13:51.954213 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:13:51.954182    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:13:52.025189 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:13:52.025152    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a981a8dd-5655-4d2d-9a6f-2ffa0eca0496-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-8c5sj\" (UID: \"a981a8dd-5655-4d2d-9a6f-2ffa0eca0496\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-8c5sj"
Jul 30 17:13:52.025189 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:13:52.025192    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5czn\" (UniqueName: \"kubernetes.io/projected/a981a8dd-5655-4d2d-9a6f-2ffa0eca0496-kube-api-access-j5czn\") pod \"message-dumper-predictor-6575bcfdfd-8c5sj\" (UID: \"a981a8dd-5655-4d2d-9a6f-2ffa0eca0496\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-8c5sj"
Jul 30 17:13:52.025393 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:13:52.025228    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a981a8dd-5655-4d2d-9a6f-2ffa0eca0496-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-8c5sj\" (UID: \"a981a8dd-5655-4d2d-9a6f-2ffa0eca0496\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-8c5sj"
Jul 30 17:13:52.025946 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:13:52.025921    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a981a8dd-5655-4d2d-9a6f-2ffa0eca0496-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-8c5sj\" (UID: \"a981a8dd-5655-4d2d-9a6f-2ffa0eca0496\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-8c5sj"
Jul 30 17:13:52.027675 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:13:52.027653    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a981a8dd-5655-4d2d-9a6f-2ffa0eca0496-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-8c5sj\" (UID: \"a981a8dd-5655-4d2d-9a6f-2ffa0eca0496\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-8c5sj"
Jul 30 17:13:52.034739 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:13:52.034717    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5czn\" (UniqueName: \"kubernetes.io/projected/a981a8dd-5655-4d2d-9a6f-2ffa0eca0496-kube-api-access-j5czn\") pod \"message-dumper-predictor-6575bcfdfd-8c5sj\" (UID: \"a981a8dd-5655-4d2d-9a6f-2ffa0eca0496\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-8c5sj"
Jul 30 17:13:52.215163 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:13:52.215075    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-8c5sj"
Jul 30 17:13:52.344704 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:13:52.344652    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-8c5sj"]
Jul 30 17:13:52.347206 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:13:52.347173    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda981a8dd_5655_4d2d_9a6f_2ffa0eca0496.slice/crio-53c2b96d0abe9239b63af5c51cd35b3a7355b8397b5f4474e4a4beb68415306e WatchSource:0}: Error finding container 53c2b96d0abe9239b63af5c51cd35b3a7355b8397b5f4474e4a4beb68415306e: Status 404 returned error can't find the container with id 53c2b96d0abe9239b63af5c51cd35b3a7355b8397b5f4474e4a4beb68415306e
Jul 30 17:13:52.349182 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:13:52.349160    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 30 17:13:52.707650 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:13:52.707611    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-8c5sj" event={"ID":"a981a8dd-5655-4d2d-9a6f-2ffa0eca0496","Type":"ContainerStarted","Data":"53c2b96d0abe9239b63af5c51cd35b3a7355b8397b5f4474e4a4beb68415306e"}
Jul 30 17:13:53.713479 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:13:53.713452    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-8c5sj" event={"ID":"a981a8dd-5655-4d2d-9a6f-2ffa0eca0496","Type":"ContainerStarted","Data":"2fe787dc47218b5d801b8eb25f38d27b5b281b41b067936753c330618e7ddf9c"}
Jul 30 17:13:56.728939 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:13:56.728904    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-8c5sj" event={"ID":"a981a8dd-5655-4d2d-9a6f-2ffa0eca0496","Type":"ContainerStarted","Data":"4c7e86e6351cf4723f99ad85c8715797d89992d0e95abcc95c23042b31854a65"}
Jul 30 17:13:56.729456 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:13:56.729122    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-8c5sj"
Jul 30 17:13:56.729456 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:13:56.729278    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-8c5sj"
Jul 30 17:13:56.730996 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:13:56.730980    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-8c5sj"
Jul 30 17:13:56.759351 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:13:56.759302    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-8c5sj" podStartSLOduration=2.209613067 podStartE2EDuration="5.759288042s" totalImagesPullingTime="3.549674975s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:13:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:13:56.758322335 +0000 UTC m=+811.245689736" watchObservedRunningTime="2026-07-30 17:13:56.759288042 +0000 UTC m=+811.246655443"
Jul 30 17:14:03.742821 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:14:03.742795    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-8c5sj"
Jul 30 17:14:11.902729 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:14:11.902698    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:14:12.049623 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:14:12.049591    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:15:41.351731 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:15:41.351702    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:15:46.970762 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:15:46.970730    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-predictor-6575bcfdfd-8c5sj_a981a8dd-5655-4d2d-9a6f-2ffa0eca0496/kserve-container/0.log"
Jul 30 17:15:47.190959 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:15:47.190922    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:15:47.269064 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:15:47.268969    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-8c5sj"]
Jul 30 17:15:47.269297 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:15:47.269268    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-8c5sj" podUID="a981a8dd-5655-4d2d-9a6f-2ffa0eca0496" containerName="kserve-container" containerID="cri-o://2fe787dc47218b5d801b8eb25f38d27b5b281b41b067936753c330618e7ddf9c" gracePeriod=30
Jul 30 17:15:47.269297 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:15:47.269285    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-8c5sj" podUID="a981a8dd-5655-4d2d-9a6f-2ffa0eca0496" containerName="kube-rbac-proxy" containerID="cri-o://4c7e86e6351cf4723f99ad85c8715797d89992d0e95abcc95c23042b31854a65" gracePeriod=30
Jul 30 17:15:47.282218 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:15:47.282192    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:15:47.353323 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:15:47.353291    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:15:47.521520 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:15:47.521465    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-8c5sj"
Jul 30 17:15:47.613548 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:15:47.613521    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j5czn\" (UniqueName: \"kubernetes.io/projected/a981a8dd-5655-4d2d-9a6f-2ffa0eca0496-kube-api-access-j5czn\") pod \"a981a8dd-5655-4d2d-9a6f-2ffa0eca0496\" (UID: \"a981a8dd-5655-4d2d-9a6f-2ffa0eca0496\") "
Jul 30 17:15:47.613711 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:15:47.613576    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a981a8dd-5655-4d2d-9a6f-2ffa0eca0496-kube-rbac-proxy-sar-config\") pod \"a981a8dd-5655-4d2d-9a6f-2ffa0eca0496\" (UID: \"a981a8dd-5655-4d2d-9a6f-2ffa0eca0496\") "
Jul 30 17:15:47.613711 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:15:47.613604    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a981a8dd-5655-4d2d-9a6f-2ffa0eca0496-proxy-tls\") pod \"a981a8dd-5655-4d2d-9a6f-2ffa0eca0496\" (UID: \"a981a8dd-5655-4d2d-9a6f-2ffa0eca0496\") "
Jul 30 17:15:47.613940 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:15:47.613916    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a981a8dd-5655-4d2d-9a6f-2ffa0eca0496-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a981a8dd-5655-4d2d-9a6f-2ffa0eca0496" (UID: "a981a8dd-5655-4d2d-9a6f-2ffa0eca0496"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:15:47.615761 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:15:47.615728    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a981a8dd-5655-4d2d-9a6f-2ffa0eca0496-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a981a8dd-5655-4d2d-9a6f-2ffa0eca0496" (UID: "a981a8dd-5655-4d2d-9a6f-2ffa0eca0496"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:15:47.615879 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:15:47.615789    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a981a8dd-5655-4d2d-9a6f-2ffa0eca0496-kube-api-access-j5czn" (OuterVolumeSpecName: "kube-api-access-j5czn") pod "a981a8dd-5655-4d2d-9a6f-2ffa0eca0496" (UID: "a981a8dd-5655-4d2d-9a6f-2ffa0eca0496"). InnerVolumeSpecName "kube-api-access-j5czn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 17:15:47.714194 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:15:47.714163    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5czn\" (UniqueName: \"kubernetes.io/projected/a981a8dd-5655-4d2d-9a6f-2ffa0eca0496-kube-api-access-j5czn\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:15:47.714194 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:15:47.714190    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a981a8dd-5655-4d2d-9a6f-2ffa0eca0496-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:15:47.714194 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:15:47.714200    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a981a8dd-5655-4d2d-9a6f-2ffa0eca0496-proxy-tls\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:15:47.746353 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:15:47.746324    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:15:48.247926 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:15:48.247895    2555 generic.go:356] "Generic (PLEG): container finished" podID="a981a8dd-5655-4d2d-9a6f-2ffa0eca0496" containerID="4c7e86e6351cf4723f99ad85c8715797d89992d0e95abcc95c23042b31854a65" exitCode=2
Jul 30 17:15:48.247926 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:15:48.247920    2555 generic.go:356] "Generic (PLEG): container finished" podID="a981a8dd-5655-4d2d-9a6f-2ffa0eca0496" containerID="2fe787dc47218b5d801b8eb25f38d27b5b281b41b067936753c330618e7ddf9c" exitCode=2
Jul 30 17:15:48.248426 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:15:48.247940    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-8c5sj"
Jul 30 17:15:48.248426 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:15:48.247977    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-8c5sj" event={"ID":"a981a8dd-5655-4d2d-9a6f-2ffa0eca0496","Type":"ContainerDied","Data":"4c7e86e6351cf4723f99ad85c8715797d89992d0e95abcc95c23042b31854a65"}
Jul 30 17:15:48.248426 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:15:48.248045    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-8c5sj" event={"ID":"a981a8dd-5655-4d2d-9a6f-2ffa0eca0496","Type":"ContainerDied","Data":"2fe787dc47218b5d801b8eb25f38d27b5b281b41b067936753c330618e7ddf9c"}
Jul 30 17:15:48.248426 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:15:48.248067    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-8c5sj" event={"ID":"a981a8dd-5655-4d2d-9a6f-2ffa0eca0496","Type":"ContainerDied","Data":"53c2b96d0abe9239b63af5c51cd35b3a7355b8397b5f4474e4a4beb68415306e"}
Jul 30 17:15:48.248426 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:15:48.248092    2555 scope.go:117] "RemoveContainer" containerID="4c7e86e6351cf4723f99ad85c8715797d89992d0e95abcc95c23042b31854a65"
Jul 30 17:15:48.256717 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:15:48.256696    2555 scope.go:117] "RemoveContainer" containerID="2fe787dc47218b5d801b8eb25f38d27b5b281b41b067936753c330618e7ddf9c"
Jul 30 17:15:48.268340 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:15:48.268313    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-8c5sj"]
Jul 30 17:15:48.271698 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:15:48.271502    2555 scope.go:117] "RemoveContainer" containerID="4c7e86e6351cf4723f99ad85c8715797d89992d0e95abcc95c23042b31854a65"
Jul 30 17:15:48.271797 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:15:48.271777    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c7e86e6351cf4723f99ad85c8715797d89992d0e95abcc95c23042b31854a65\": container with ID starting with 4c7e86e6351cf4723f99ad85c8715797d89992d0e95abcc95c23042b31854a65 not found: ID does not exist" containerID="4c7e86e6351cf4723f99ad85c8715797d89992d0e95abcc95c23042b31854a65"
Jul 30 17:15:48.271860 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:15:48.271805    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c7e86e6351cf4723f99ad85c8715797d89992d0e95abcc95c23042b31854a65"} err="failed to get container status \"4c7e86e6351cf4723f99ad85c8715797d89992d0e95abcc95c23042b31854a65\": rpc error: code = NotFound desc = could not find container \"4c7e86e6351cf4723f99ad85c8715797d89992d0e95abcc95c23042b31854a65\": container with ID starting with 4c7e86e6351cf4723f99ad85c8715797d89992d0e95abcc95c23042b31854a65 not found: ID does not exist"
Jul 30 17:15:48.271860 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:15:48.271823    2555 scope.go:117] "RemoveContainer" containerID="2fe787dc47218b5d801b8eb25f38d27b5b281b41b067936753c330618e7ddf9c"
Jul 30 17:15:48.272085 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:15:48.272063    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fe787dc47218b5d801b8eb25f38d27b5b281b41b067936753c330618e7ddf9c\": container with ID starting with 2fe787dc47218b5d801b8eb25f38d27b5b281b41b067936753c330618e7ddf9c not found: ID does not exist" containerID="2fe787dc47218b5d801b8eb25f38d27b5b281b41b067936753c330618e7ddf9c"
Jul 30 17:15:48.272154 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:15:48.272091    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fe787dc47218b5d801b8eb25f38d27b5b281b41b067936753c330618e7ddf9c"} err="failed to get container status \"2fe787dc47218b5d801b8eb25f38d27b5b281b41b067936753c330618e7ddf9c\": rpc error: code = NotFound desc = could not find container \"2fe787dc47218b5d801b8eb25f38d27b5b281b41b067936753c330618e7ddf9c\": container with ID starting with 2fe787dc47218b5d801b8eb25f38d27b5b281b41b067936753c330618e7ddf9c not found: ID does not exist"
Jul 30 17:15:48.272154 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:15:48.272106    2555 scope.go:117] "RemoveContainer" containerID="4c7e86e6351cf4723f99ad85c8715797d89992d0e95abcc95c23042b31854a65"
Jul 30 17:15:48.272353 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:15:48.272335    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c7e86e6351cf4723f99ad85c8715797d89992d0e95abcc95c23042b31854a65"} err="failed to get container status \"4c7e86e6351cf4723f99ad85c8715797d89992d0e95abcc95c23042b31854a65\": rpc error: code = NotFound desc = could not find container \"4c7e86e6351cf4723f99ad85c8715797d89992d0e95abcc95c23042b31854a65\": container with ID starting with 4c7e86e6351cf4723f99ad85c8715797d89992d0e95abcc95c23042b31854a65 not found: ID does not exist"
Jul 30 17:15:48.272421 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:15:48.272353    2555 scope.go:117] "RemoveContainer" containerID="2fe787dc47218b5d801b8eb25f38d27b5b281b41b067936753c330618e7ddf9c"
Jul 30 17:15:48.272580 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:15:48.272561    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fe787dc47218b5d801b8eb25f38d27b5b281b41b067936753c330618e7ddf9c"} err="failed to get container status \"2fe787dc47218b5d801b8eb25f38d27b5b281b41b067936753c330618e7ddf9c\": rpc error: code = NotFound desc = could not find container \"2fe787dc47218b5d801b8eb25f38d27b5b281b41b067936753c330618e7ddf9c\": container with ID starting with 2fe787dc47218b5d801b8eb25f38d27b5b281b41b067936753c330618e7ddf9c not found: ID does not exist"
Jul 30 17:15:48.272746 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:15:48.272727    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-8c5sj"]
Jul 30 17:15:50.060361 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:15:50.060329    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a981a8dd-5655-4d2d-9a6f-2ffa0eca0496" path="/var/lib/kubelet/pods/a981a8dd-5655-4d2d-9a6f-2ffa0eca0496/volumes"
Jul 30 17:16:58.446417 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:16:58.446335    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:17:54.217489 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:17:54.217437    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:18:43.540712 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:18:43.540674    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:18:48.619201 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:18:48.619166    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:18:48.718316 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:18:48.718285    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:21:04.995990 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:21:04.995957    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:21:10.070120 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:21:10.070090    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:21:10.139336 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:21:10.139301    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:23:16.814383 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:23:16.814302    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:23:21.891117 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:23:21.891083    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:23:22.343000 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:23:22.342972    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:23:32.520710 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:23:32.520674    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:25:09.020205 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:25:09.020162    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:25:13.325988 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:25:13.325956    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:25:13.638961 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:25:13.638874    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:25:15.522495 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:25:15.522450    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:25:22.092378 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:25:22.092337    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:25:55.623734 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:25:55.623645    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:26:00.736351 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:26:00.736315    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:26:01.031207 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:26:01.031129    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:26:35.984718 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:26:35.984680    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:26:41.065420 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:26:41.065384    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:26:41.146428 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:26:41.146385    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:26:41.205260 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:26:41.205225    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:28:11.254191 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:28:11.254157    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:28:11.314456 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:28:11.314415    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:28:16.339102 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:28:16.339068    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:28:16.406421 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:28:16.406393    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:28:16.488267 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:28:16.488234    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:28:16.812292 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:28:16.812246    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:29:46.502899 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:29:46.502859    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:29:46.542822 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:29:46.542784    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-2bfdd"]
Jul 30 17:29:46.543238 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:29:46.543220    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a981a8dd-5655-4d2d-9a6f-2ffa0eca0496" containerName="kube-rbac-proxy"
Jul 30 17:29:46.543336 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:29:46.543240    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a981a8dd-5655-4d2d-9a6f-2ffa0eca0496" containerName="kube-rbac-proxy"
Jul 30 17:29:46.543336 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:29:46.543253    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a981a8dd-5655-4d2d-9a6f-2ffa0eca0496" containerName="kserve-container"
Jul 30 17:29:46.543336 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:29:46.543261    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a981a8dd-5655-4d2d-9a6f-2ffa0eca0496" containerName="kserve-container"
Jul 30 17:29:46.543489 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:29:46.543378    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a981a8dd-5655-4d2d-9a6f-2ffa0eca0496" containerName="kube-rbac-proxy"
Jul 30 17:29:46.543489 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:29:46.543394    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a981a8dd-5655-4d2d-9a6f-2ffa0eca0496" containerName="kserve-container"
Jul 30 17:29:46.546615 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:29:46.546591    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-2bfdd"
Jul 30 17:29:46.549063 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:29:46.549045    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Jul 30 17:29:46.549173 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:29:46.549143    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-plw5t\""
Jul 30 17:29:46.549254 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:29:46.549236    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 30 17:29:46.550166 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:29:46.550151    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 30 17:29:46.550166 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:29:46.550153    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Jul 30 17:29:46.550303 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:29:46.550198    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 30 17:29:46.550303 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:29:46.550300    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 30 17:29:46.556629 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:29:46.556608    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-2bfdd"]
Jul 30 17:29:46.630157 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:29:46.630124    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-426z6\" (UniqueName: \"kubernetes.io/projected/c385abcb-2026-4b28-a0e2-1beb879f8082-kube-api-access-426z6\") pod \"isvc-canary-promote-v2-predictor-66f6768f84-2bfdd\" (UID: \"c385abcb-2026-4b28-a0e2-1beb879f8082\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-2bfdd"
Jul 30 17:29:46.630340 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:29:46.630168    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c385abcb-2026-4b28-a0e2-1beb879f8082-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-66f6768f84-2bfdd\" (UID: \"c385abcb-2026-4b28-a0e2-1beb879f8082\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-2bfdd"
Jul 30 17:29:46.630340 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:29:46.630191    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c385abcb-2026-4b28-a0e2-1beb879f8082-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-66f6768f84-2bfdd\" (UID: \"c385abcb-2026-4b28-a0e2-1beb879f8082\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-2bfdd"
Jul 30 17:29:46.630340 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:29:46.630265    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c385abcb-2026-4b28-a0e2-1beb879f8082-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-66f6768f84-2bfdd\" (UID: \"c385abcb-2026-4b28-a0e2-1beb879f8082\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-2bfdd"
Jul 30 17:29:46.630340 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:29:46.630323    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c385abcb-2026-4b28-a0e2-1beb879f8082-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-66f6768f84-2bfdd\" (UID: \"c385abcb-2026-4b28-a0e2-1beb879f8082\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-2bfdd"
Jul 30 17:29:46.731411 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:29:46.731375    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c385abcb-2026-4b28-a0e2-1beb879f8082-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-66f6768f84-2bfdd\" (UID: \"c385abcb-2026-4b28-a0e2-1beb879f8082\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-2bfdd"
Jul 30 17:29:46.731613 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:29:46.731451    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-426z6\" (UniqueName: \"kubernetes.io/projected/c385abcb-2026-4b28-a0e2-1beb879f8082-kube-api-access-426z6\") pod \"isvc-canary-promote-v2-predictor-66f6768f84-2bfdd\" (UID: \"c385abcb-2026-4b28-a0e2-1beb879f8082\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-2bfdd"
Jul 30 17:29:46.731613 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:29:46.731484    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c385abcb-2026-4b28-a0e2-1beb879f8082-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-66f6768f84-2bfdd\" (UID: \"c385abcb-2026-4b28-a0e2-1beb879f8082\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-2bfdd"
Jul 30 17:29:46.731613 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:29:46.731504    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c385abcb-2026-4b28-a0e2-1beb879f8082-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-66f6768f84-2bfdd\" (UID: \"c385abcb-2026-4b28-a0e2-1beb879f8082\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-2bfdd"
Jul 30 17:29:46.731613 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:29:46.731524    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c385abcb-2026-4b28-a0e2-1beb879f8082-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-66f6768f84-2bfdd\" (UID: \"c385abcb-2026-4b28-a0e2-1beb879f8082\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-2bfdd"
Jul 30 17:29:46.731970 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:29:46.731948    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c385abcb-2026-4b28-a0e2-1beb879f8082-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-66f6768f84-2bfdd\" (UID: \"c385abcb-2026-4b28-a0e2-1beb879f8082\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-2bfdd"
Jul 30 17:29:46.732173 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:29:46.732154    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c385abcb-2026-4b28-a0e2-1beb879f8082-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-66f6768f84-2bfdd\" (UID: \"c385abcb-2026-4b28-a0e2-1beb879f8082\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-2bfdd"
Jul 30 17:29:46.732232 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:29:46.732177    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c385abcb-2026-4b28-a0e2-1beb879f8082-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-66f6768f84-2bfdd\" (UID: \"c385abcb-2026-4b28-a0e2-1beb879f8082\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-2bfdd"
Jul 30 17:29:46.733867 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:29:46.733850    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c385abcb-2026-4b28-a0e2-1beb879f8082-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-66f6768f84-2bfdd\" (UID: \"c385abcb-2026-4b28-a0e2-1beb879f8082\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-2bfdd"
Jul 30 17:29:46.741188 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:29:46.741165    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-426z6\" (UniqueName: \"kubernetes.io/projected/c385abcb-2026-4b28-a0e2-1beb879f8082-kube-api-access-426z6\") pod \"isvc-canary-promote-v2-predictor-66f6768f84-2bfdd\" (UID: \"c385abcb-2026-4b28-a0e2-1beb879f8082\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-2bfdd"
Jul 30 17:29:46.857920 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:29:46.857886    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-2bfdd"
Jul 30 17:29:46.990315 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:29:46.990285    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-2bfdd"]
Jul 30 17:29:46.991457 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:29:46.991434    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc385abcb_2026_4b28_a0e2_1beb879f8082.slice/crio-4a68076853640218d8c71efebfa8856116ec255450c2b720b071ab76cc8f89e6 WatchSource:0}: Error finding container 4a68076853640218d8c71efebfa8856116ec255450c2b720b071ab76cc8f89e6: Status 404 returned error can't find the container with id 4a68076853640218d8c71efebfa8856116ec255450c2b720b071ab76cc8f89e6
Jul 30 17:29:46.993599 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:29:46.993579    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 30 17:29:46.999647 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:29:46.999623    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-2bfdd" event={"ID":"c385abcb-2026-4b28-a0e2-1beb879f8082","Type":"ContainerStarted","Data":"4a68076853640218d8c71efebfa8856116ec255450c2b720b071ab76cc8f89e6"}
Jul 30 17:29:51.023122 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:29:51.023035    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-2bfdd" event={"ID":"c385abcb-2026-4b28-a0e2-1beb879f8082","Type":"ContainerStarted","Data":"4b7b6e268aa70b513d2b19c1be926551c371abda1d7eb25b1c39f1890fcb8067"}
Jul 30 17:29:55.042955 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:29:55.042925    2555 generic.go:356] "Generic (PLEG): container finished" podID="c385abcb-2026-4b28-a0e2-1beb879f8082" containerID="4b7b6e268aa70b513d2b19c1be926551c371abda1d7eb25b1c39f1890fcb8067" exitCode=0
Jul 30 17:29:55.043338 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:29:55.042999    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-2bfdd" event={"ID":"c385abcb-2026-4b28-a0e2-1beb879f8082","Type":"ContainerDied","Data":"4b7b6e268aa70b513d2b19c1be926551c371abda1d7eb25b1c39f1890fcb8067"}
Jul 30 17:30:09.125394 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:30:09.125356    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-2bfdd" event={"ID":"c385abcb-2026-4b28-a0e2-1beb879f8082","Type":"ContainerStarted","Data":"eb856086e65d8043aa68eb2af9d32e07c17b40e56f7ebda75d05a62e8a732dd9"}
Jul 30 17:30:09.125751 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:30:09.125442    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-2bfdd" event={"ID":"c385abcb-2026-4b28-a0e2-1beb879f8082","Type":"ContainerStarted","Data":"1a4df3ca0faa03eea503c4829f5acf3897c33ce7c2a5ab621de557a36bd273d2"}
Jul 30 17:30:09.125751 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:30:09.125642    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-2bfdd"
Jul 30 17:30:09.145202 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:30:09.145150    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-2bfdd" podStartSLOduration=2.62267271 podStartE2EDuration="23.145134905s" totalImagesPullingTime="16.707597316s" totalInitContainerRuntime="3.814864879s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:29:46 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:30:09.144370584 +0000 UTC m=+1783.631737986" watchObservedRunningTime="2026-07-30 17:30:09.145134905 +0000 UTC m=+1783.632502307"
Jul 30 17:30:10.130405 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:30:10.130380    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-2bfdd"
Jul 30 17:30:10.131743 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:30:10.131700    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-2bfdd" podUID="c385abcb-2026-4b28-a0e2-1beb879f8082" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 30 17:30:11.134404 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:30:11.134368    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-2bfdd" podUID="c385abcb-2026-4b28-a0e2-1beb879f8082" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 30 17:30:16.138781 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:30:16.138752    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-2bfdd"
Jul 30 17:30:16.139326 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:30:16.139302    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-2bfdd" podUID="c385abcb-2026-4b28-a0e2-1beb879f8082" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 30 17:30:26.139490 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:30:26.139453    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-2bfdd" podUID="c385abcb-2026-4b28-a0e2-1beb879f8082" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 30 17:30:36.139631 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:30:36.139589    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-2bfdd" podUID="c385abcb-2026-4b28-a0e2-1beb879f8082" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 30 17:30:46.140226 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:30:46.140185    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-2bfdd" podUID="c385abcb-2026-4b28-a0e2-1beb879f8082" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 30 17:30:56.139705 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:30:56.139665    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-2bfdd" podUID="c385abcb-2026-4b28-a0e2-1beb879f8082" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 30 17:31:06.139721 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:06.139677    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-2bfdd" podUID="c385abcb-2026-4b28-a0e2-1beb879f8082" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 30 17:31:16.139379 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:16.139337    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-2bfdd" podUID="c385abcb-2026-4b28-a0e2-1beb879f8082" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 30 17:31:26.140097 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:26.140070    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-2bfdd"
Jul 30 17:31:26.857205 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:26.857170    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-2bfdd"]
Jul 30 17:31:26.857480 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:26.857429    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-2bfdd" podUID="c385abcb-2026-4b28-a0e2-1beb879f8082" containerName="kserve-container" containerID="cri-o://eb856086e65d8043aa68eb2af9d32e07c17b40e56f7ebda75d05a62e8a732dd9" gracePeriod=30
Jul 30 17:31:26.857643 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:26.857468    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-2bfdd" podUID="c385abcb-2026-4b28-a0e2-1beb879f8082" containerName="kube-rbac-proxy" containerID="cri-o://1a4df3ca0faa03eea503c4829f5acf3897c33ce7c2a5ab621de557a36bd273d2" gracePeriod=30
Jul 30 17:31:26.893543 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:26.893518    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:31:27.503253 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:27.503223    2555 generic.go:356] "Generic (PLEG): container finished" podID="c385abcb-2026-4b28-a0e2-1beb879f8082" containerID="1a4df3ca0faa03eea503c4829f5acf3897c33ce7c2a5ab621de557a36bd273d2" exitCode=2
Jul 30 17:31:27.503627 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:27.503297    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-2bfdd" event={"ID":"c385abcb-2026-4b28-a0e2-1beb879f8082","Type":"ContainerDied","Data":"1a4df3ca0faa03eea503c4829f5acf3897c33ce7c2a5ab621de557a36bd273d2"}
Jul 30 17:31:31.134556 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:31.134516    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-2bfdd" podUID="c385abcb-2026-4b28-a0e2-1beb879f8082" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Jul 30 17:31:31.523404 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:31.523372    2555 generic.go:356] "Generic (PLEG): container finished" podID="c385abcb-2026-4b28-a0e2-1beb879f8082" containerID="eb856086e65d8043aa68eb2af9d32e07c17b40e56f7ebda75d05a62e8a732dd9" exitCode=0
Jul 30 17:31:31.523559 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:31.523455    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-2bfdd" event={"ID":"c385abcb-2026-4b28-a0e2-1beb879f8082","Type":"ContainerDied","Data":"eb856086e65d8043aa68eb2af9d32e07c17b40e56f7ebda75d05a62e8a732dd9"}
Jul 30 17:31:31.603372 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:31.603348    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-2bfdd"
Jul 30 17:31:31.753908 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:31.753834    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c385abcb-2026-4b28-a0e2-1beb879f8082-cabundle-cert\") pod \"c385abcb-2026-4b28-a0e2-1beb879f8082\" (UID: \"c385abcb-2026-4b28-a0e2-1beb879f8082\") "
Jul 30 17:31:31.753908 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:31.753903    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-426z6\" (UniqueName: \"kubernetes.io/projected/c385abcb-2026-4b28-a0e2-1beb879f8082-kube-api-access-426z6\") pod \"c385abcb-2026-4b28-a0e2-1beb879f8082\" (UID: \"c385abcb-2026-4b28-a0e2-1beb879f8082\") "
Jul 30 17:31:31.754107 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:31.753941    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c385abcb-2026-4b28-a0e2-1beb879f8082-kube-rbac-proxy-sar-config\") pod \"c385abcb-2026-4b28-a0e2-1beb879f8082\" (UID: \"c385abcb-2026-4b28-a0e2-1beb879f8082\") "
Jul 30 17:31:31.754107 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:31.753986    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c385abcb-2026-4b28-a0e2-1beb879f8082-kserve-provision-location\") pod \"c385abcb-2026-4b28-a0e2-1beb879f8082\" (UID: \"c385abcb-2026-4b28-a0e2-1beb879f8082\") "
Jul 30 17:31:31.754107 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:31.754037    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c385abcb-2026-4b28-a0e2-1beb879f8082-proxy-tls\") pod \"c385abcb-2026-4b28-a0e2-1beb879f8082\" (UID: \"c385abcb-2026-4b28-a0e2-1beb879f8082\") "
Jul 30 17:31:31.754290 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:31.754265    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c385abcb-2026-4b28-a0e2-1beb879f8082-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c385abcb-2026-4b28-a0e2-1beb879f8082" (UID: "c385abcb-2026-4b28-a0e2-1beb879f8082"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 17:31:31.754329 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:31.754307    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c385abcb-2026-4b28-a0e2-1beb879f8082-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c385abcb-2026-4b28-a0e2-1beb879f8082" (UID: "c385abcb-2026-4b28-a0e2-1beb879f8082"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:31:31.754362 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:31.754331    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c385abcb-2026-4b28-a0e2-1beb879f8082-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c385abcb-2026-4b28-a0e2-1beb879f8082" (UID: "c385abcb-2026-4b28-a0e2-1beb879f8082"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:31:31.756103 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:31.756077    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c385abcb-2026-4b28-a0e2-1beb879f8082-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c385abcb-2026-4b28-a0e2-1beb879f8082" (UID: "c385abcb-2026-4b28-a0e2-1beb879f8082"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:31:31.756215 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:31.756171    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c385abcb-2026-4b28-a0e2-1beb879f8082-kube-api-access-426z6" (OuterVolumeSpecName: "kube-api-access-426z6") pod "c385abcb-2026-4b28-a0e2-1beb879f8082" (UID: "c385abcb-2026-4b28-a0e2-1beb879f8082"). InnerVolumeSpecName "kube-api-access-426z6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 17:31:31.855532 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:31.855496    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c385abcb-2026-4b28-a0e2-1beb879f8082-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:31:31.855532 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:31.855535    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c385abcb-2026-4b28-a0e2-1beb879f8082-kserve-provision-location\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:31:31.855728 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:31.855552    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c385abcb-2026-4b28-a0e2-1beb879f8082-proxy-tls\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:31:31.855728 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:31.855567    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c385abcb-2026-4b28-a0e2-1beb879f8082-cabundle-cert\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:31:31.855728 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:31.855582    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-426z6\" (UniqueName: \"kubernetes.io/projected/c385abcb-2026-4b28-a0e2-1beb879f8082-kube-api-access-426z6\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:31:31.872894 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:31.872861    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:31:31.946434 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:31.946401    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-4glj8"]
Jul 30 17:31:31.946768 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:31.946755    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c385abcb-2026-4b28-a0e2-1beb879f8082" containerName="storage-initializer"
Jul 30 17:31:31.946815 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:31.946769    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c385abcb-2026-4b28-a0e2-1beb879f8082" containerName="storage-initializer"
Jul 30 17:31:31.946815 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:31.946783    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c385abcb-2026-4b28-a0e2-1beb879f8082" containerName="kserve-container"
Jul 30 17:31:31.946815 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:31.946788    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c385abcb-2026-4b28-a0e2-1beb879f8082" containerName="kserve-container"
Jul 30 17:31:31.946815 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:31.946810    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c385abcb-2026-4b28-a0e2-1beb879f8082" containerName="kube-rbac-proxy"
Jul 30 17:31:31.946815 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:31.946816    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c385abcb-2026-4b28-a0e2-1beb879f8082" containerName="kube-rbac-proxy"
Jul 30 17:31:31.946980 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:31.946871    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c385abcb-2026-4b28-a0e2-1beb879f8082" containerName="kube-rbac-proxy"
Jul 30 17:31:31.946980 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:31.946886    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c385abcb-2026-4b28-a0e2-1beb879f8082" containerName="kserve-container"
Jul 30 17:31:31.949325 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:31.949306    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-4glj8"
Jul 30 17:31:31.952035 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:31.951992    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Jul 30 17:31:31.952163 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:31.952101    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Jul 30 17:31:31.962405 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:31.962383    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-jkpgt"]
Jul 30 17:31:31.972796 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:31.972772    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-4glj8"]
Jul 30 17:31:31.972925 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:31.972869    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-jkpgt"
Jul 30 17:31:31.975521 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:31.975485    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Jul 30 17:31:31.989284 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:31.989258    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-jkpgt"]
Jul 30 17:31:31.992036 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:31.991987    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:31:32.051429 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:32.051391    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:31:32.056755 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:32.056734    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz9dr\" (UniqueName: \"kubernetes.io/projected/0ed95623-8c1a-430c-bcae-6ac695c65e17-kube-api-access-fz9dr\") pod \"isvc-canary-rollback-v2-predictor-578c74fc8-jkpgt\" (UID: \"0ed95623-8c1a-430c-bcae-6ac695c65e17\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-jkpgt"
Jul 30 17:31:32.056862 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:32.056807    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6a88b6a-7228-455f-aa94-57feda298014-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7c9c87bcff-4glj8\" (UID: \"d6a88b6a-7228-455f-aa94-57feda298014\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-4glj8"
Jul 30 17:31:32.056917 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:32.056865    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6a88b6a-7228-455f-aa94-57feda298014-proxy-tls\") pod \"isvc-canary-rollback-predictor-7c9c87bcff-4glj8\" (UID: \"d6a88b6a-7228-455f-aa94-57feda298014\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-4glj8"
Jul 30 17:31:32.057055 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:32.057037    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0ed95623-8c1a-430c-bcae-6ac695c65e17-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-578c74fc8-jkpgt\" (UID: \"0ed95623-8c1a-430c-bcae-6ac695c65e17\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-jkpgt"
Jul 30 17:31:32.057131 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:32.057071    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6a88b6a-7228-455f-aa94-57feda298014-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7c9c87bcff-4glj8\" (UID: \"d6a88b6a-7228-455f-aa94-57feda298014\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-4glj8"
Jul 30 17:31:32.057131 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:32.057111    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0ed95623-8c1a-430c-bcae-6ac695c65e17-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-578c74fc8-jkpgt\" (UID: \"0ed95623-8c1a-430c-bcae-6ac695c65e17\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-jkpgt"
Jul 30 17:31:32.057236 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:32.057139    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfq8z\" (UniqueName: \"kubernetes.io/projected/d6a88b6a-7228-455f-aa94-57feda298014-kube-api-access-rfq8z\") pod \"isvc-canary-rollback-predictor-7c9c87bcff-4glj8\" (UID: \"d6a88b6a-7228-455f-aa94-57feda298014\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-4glj8"
Jul 30 17:31:32.057236 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:32.057176    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6a88b6a-7228-455f-aa94-57feda298014-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7c9c87bcff-4glj8\" (UID: \"d6a88b6a-7228-455f-aa94-57feda298014\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-4glj8"
Jul 30 17:31:32.057236 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:32.057200    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0ed95623-8c1a-430c-bcae-6ac695c65e17-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-578c74fc8-jkpgt\" (UID: \"0ed95623-8c1a-430c-bcae-6ac695c65e17\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-jkpgt"
Jul 30 17:31:32.057377 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:32.057238    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0ed95623-8c1a-430c-bcae-6ac695c65e17-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-578c74fc8-jkpgt\" (UID: \"0ed95623-8c1a-430c-bcae-6ac695c65e17\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-jkpgt"
Jul 30 17:31:32.154122 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:32.154094    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:31:32.158342 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:32.158306    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6a88b6a-7228-455f-aa94-57feda298014-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7c9c87bcff-4glj8\" (UID: \"d6a88b6a-7228-455f-aa94-57feda298014\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-4glj8"
Jul 30 17:31:32.158451 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:32.158363    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0ed95623-8c1a-430c-bcae-6ac695c65e17-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-578c74fc8-jkpgt\" (UID: \"0ed95623-8c1a-430c-bcae-6ac695c65e17\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-jkpgt"
Jul 30 17:31:32.158451 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:32.158393    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfq8z\" (UniqueName: \"kubernetes.io/projected/d6a88b6a-7228-455f-aa94-57feda298014-kube-api-access-rfq8z\") pod \"isvc-canary-rollback-predictor-7c9c87bcff-4glj8\" (UID: \"d6a88b6a-7228-455f-aa94-57feda298014\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-4glj8"
Jul 30 17:31:32.158451 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:32.158424    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6a88b6a-7228-455f-aa94-57feda298014-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7c9c87bcff-4glj8\" (UID: \"d6a88b6a-7228-455f-aa94-57feda298014\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-4glj8"
Jul 30 17:31:32.158451 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:32.158447    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0ed95623-8c1a-430c-bcae-6ac695c65e17-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-578c74fc8-jkpgt\" (UID: \"0ed95623-8c1a-430c-bcae-6ac695c65e17\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-jkpgt"
Jul 30 17:31:32.158694 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:32.158483    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0ed95623-8c1a-430c-bcae-6ac695c65e17-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-578c74fc8-jkpgt\" (UID: \"0ed95623-8c1a-430c-bcae-6ac695c65e17\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-jkpgt"
Jul 30 17:31:32.158694 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:32.158552    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz9dr\" (UniqueName: \"kubernetes.io/projected/0ed95623-8c1a-430c-bcae-6ac695c65e17-kube-api-access-fz9dr\") pod \"isvc-canary-rollback-v2-predictor-578c74fc8-jkpgt\" (UID: \"0ed95623-8c1a-430c-bcae-6ac695c65e17\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-jkpgt"
Jul 30 17:31:32.158694 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:32.158579    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6a88b6a-7228-455f-aa94-57feda298014-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7c9c87bcff-4glj8\" (UID: \"d6a88b6a-7228-455f-aa94-57feda298014\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-4glj8"
Jul 30 17:31:32.158694 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:32.158608    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6a88b6a-7228-455f-aa94-57feda298014-proxy-tls\") pod \"isvc-canary-rollback-predictor-7c9c87bcff-4glj8\" (UID: \"d6a88b6a-7228-455f-aa94-57feda298014\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-4glj8"
Jul 30 17:31:32.158901 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:32.158695    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0ed95623-8c1a-430c-bcae-6ac695c65e17-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-578c74fc8-jkpgt\" (UID: \"0ed95623-8c1a-430c-bcae-6ac695c65e17\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-jkpgt"
Jul 30 17:31:32.159279 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:32.159258    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6a88b6a-7228-455f-aa94-57feda298014-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7c9c87bcff-4glj8\" (UID: \"d6a88b6a-7228-455f-aa94-57feda298014\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-4glj8"
Jul 30 17:31:32.159413 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:32.159317    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0ed95623-8c1a-430c-bcae-6ac695c65e17-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-578c74fc8-jkpgt\" (UID: \"0ed95623-8c1a-430c-bcae-6ac695c65e17\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-jkpgt"
Jul 30 17:31:32.159413 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:32.159347    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0ed95623-8c1a-430c-bcae-6ac695c65e17-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-578c74fc8-jkpgt\" (UID: \"0ed95623-8c1a-430c-bcae-6ac695c65e17\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-jkpgt"
Jul 30 17:31:32.159413 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:32.159352    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6a88b6a-7228-455f-aa94-57feda298014-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7c9c87bcff-4glj8\" (UID: \"d6a88b6a-7228-455f-aa94-57feda298014\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-4glj8"
Jul 30 17:31:32.159413 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:32.159390    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6a88b6a-7228-455f-aa94-57feda298014-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7c9c87bcff-4glj8\" (UID: \"d6a88b6a-7228-455f-aa94-57feda298014\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-4glj8"
Jul 30 17:31:32.159628 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:32.159601    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0ed95623-8c1a-430c-bcae-6ac695c65e17-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-578c74fc8-jkpgt\" (UID: \"0ed95623-8c1a-430c-bcae-6ac695c65e17\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-jkpgt"
Jul 30 17:31:32.161162 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:32.161142    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0ed95623-8c1a-430c-bcae-6ac695c65e17-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-578c74fc8-jkpgt\" (UID: \"0ed95623-8c1a-430c-bcae-6ac695c65e17\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-jkpgt"
Jul 30 17:31:32.161271 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:32.161257    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6a88b6a-7228-455f-aa94-57feda298014-proxy-tls\") pod \"isvc-canary-rollback-predictor-7c9c87bcff-4glj8\" (UID: \"d6a88b6a-7228-455f-aa94-57feda298014\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-4glj8"
Jul 30 17:31:32.169449 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:32.169416    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfq8z\" (UniqueName: \"kubernetes.io/projected/d6a88b6a-7228-455f-aa94-57feda298014-kube-api-access-rfq8z\") pod \"isvc-canary-rollback-predictor-7c9c87bcff-4glj8\" (UID: \"d6a88b6a-7228-455f-aa94-57feda298014\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-4glj8"
Jul 30 17:31:32.169577 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:32.169551    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz9dr\" (UniqueName: \"kubernetes.io/projected/0ed95623-8c1a-430c-bcae-6ac695c65e17-kube-api-access-fz9dr\") pod \"isvc-canary-rollback-v2-predictor-578c74fc8-jkpgt\" (UID: \"0ed95623-8c1a-430c-bcae-6ac695c65e17\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-jkpgt"
Jul 30 17:31:32.259955 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:32.259920    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-4glj8"
Jul 30 17:31:32.283465 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:32.283437    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-jkpgt"
Jul 30 17:31:32.404788 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:32.404763    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-4glj8"]
Jul 30 17:31:32.407683 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:31:32.407656    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6a88b6a_7228_455f_aa94_57feda298014.slice/crio-1d62b971922d41286044b0f29435cc5494c27e4e6a06501468eb053c0e434be7 WatchSource:0}: Error finding container 1d62b971922d41286044b0f29435cc5494c27e4e6a06501468eb053c0e434be7: Status 404 returned error can't find the container with id 1d62b971922d41286044b0f29435cc5494c27e4e6a06501468eb053c0e434be7
Jul 30 17:31:32.433321 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:32.433301    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-jkpgt"]
Jul 30 17:31:32.443297 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:31:32.443271    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ed95623_8c1a_430c_bcae_6ac695c65e17.slice/crio-94249d8e9620df27c25d7bdd906b79279871d38db2366fdfd238375574997a3f WatchSource:0}: Error finding container 94249d8e9620df27c25d7bdd906b79279871d38db2366fdfd238375574997a3f: Status 404 returned error can't find the container with id 94249d8e9620df27c25d7bdd906b79279871d38db2366fdfd238375574997a3f
Jul 30 17:31:32.532789 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:32.532757    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-2bfdd" event={"ID":"c385abcb-2026-4b28-a0e2-1beb879f8082","Type":"ContainerDied","Data":"4a68076853640218d8c71efebfa8856116ec255450c2b720b071ab76cc8f89e6"}
Jul 30 17:31:32.532962 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:32.532806    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-2bfdd"
Jul 30 17:31:32.532962 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:32.532834    2555 scope.go:117] "RemoveContainer" containerID="1a4df3ca0faa03eea503c4829f5acf3897c33ce7c2a5ab621de557a36bd273d2"
Jul 30 17:31:32.535077 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:32.535045    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-jkpgt" event={"ID":"0ed95623-8c1a-430c-bcae-6ac695c65e17","Type":"ContainerStarted","Data":"fc1f7cf678e673a718614688d2ab015cb436fb0043538497bf18e67fd3bbd305"}
Jul 30 17:31:32.535198 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:32.535113    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-jkpgt" event={"ID":"0ed95623-8c1a-430c-bcae-6ac695c65e17","Type":"ContainerStarted","Data":"94249d8e9620df27c25d7bdd906b79279871d38db2366fdfd238375574997a3f"}
Jul 30 17:31:32.537406 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:32.537370    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-4glj8" event={"ID":"d6a88b6a-7228-455f-aa94-57feda298014","Type":"ContainerStarted","Data":"ff84720bdfa54241cfc34b54326dc3fc7af578aac791a3609ec5587a049fd26b"}
Jul 30 17:31:32.537522 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:32.537427    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-4glj8" event={"ID":"d6a88b6a-7228-455f-aa94-57feda298014","Type":"ContainerStarted","Data":"1d62b971922d41286044b0f29435cc5494c27e4e6a06501468eb053c0e434be7"}
Jul 30 17:31:32.543575 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:32.543558    2555 scope.go:117] "RemoveContainer" containerID="eb856086e65d8043aa68eb2af9d32e07c17b40e56f7ebda75d05a62e8a732dd9"
Jul 30 17:31:32.553347 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:32.553331    2555 scope.go:117] "RemoveContainer" containerID="4b7b6e268aa70b513d2b19c1be926551c371abda1d7eb25b1c39f1890fcb8067"
Jul 30 17:31:32.558730 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:32.558705    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-2bfdd"]
Jul 30 17:31:32.581080 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:32.581041    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-2bfdd"]
Jul 30 17:31:34.060941 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:34.060910    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c385abcb-2026-4b28-a0e2-1beb879f8082" path="/var/lib/kubelet/pods/c385abcb-2026-4b28-a0e2-1beb879f8082/volumes"
Jul 30 17:31:36.559628 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:36.559593    2555 generic.go:356] "Generic (PLEG): container finished" podID="d6a88b6a-7228-455f-aa94-57feda298014" containerID="ff84720bdfa54241cfc34b54326dc3fc7af578aac791a3609ec5587a049fd26b" exitCode=0
Jul 30 17:31:36.560020 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:36.559667    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-4glj8" event={"ID":"d6a88b6a-7228-455f-aa94-57feda298014","Type":"ContainerDied","Data":"ff84720bdfa54241cfc34b54326dc3fc7af578aac791a3609ec5587a049fd26b"}
Jul 30 17:31:36.561236 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:36.561214    2555 generic.go:356] "Generic (PLEG): container finished" podID="0ed95623-8c1a-430c-bcae-6ac695c65e17" containerID="fc1f7cf678e673a718614688d2ab015cb436fb0043538497bf18e67fd3bbd305" exitCode=0
Jul 30 17:31:36.561338 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:36.561253    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-jkpgt" event={"ID":"0ed95623-8c1a-430c-bcae-6ac695c65e17","Type":"ContainerDied","Data":"fc1f7cf678e673a718614688d2ab015cb436fb0043538497bf18e67fd3bbd305"}
Jul 30 17:31:37.568576 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:37.568542    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-jkpgt" event={"ID":"0ed95623-8c1a-430c-bcae-6ac695c65e17","Type":"ContainerStarted","Data":"e8bce075360f1a85d35ebd4e48652ab97ede242753dcede2c099d3bcafc32ad5"}
Jul 30 17:31:37.569241 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:37.568612    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-jkpgt" event={"ID":"0ed95623-8c1a-430c-bcae-6ac695c65e17","Type":"ContainerStarted","Data":"b8ccfcd86756e016b5010cc1add1b927ee9e0216dc588a5ed1c00ee51e315483"}
Jul 30 17:31:37.569241 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:37.568824    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-jkpgt"
Jul 30 17:31:37.570602 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:37.570582    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-4glj8" event={"ID":"d6a88b6a-7228-455f-aa94-57feda298014","Type":"ContainerStarted","Data":"9a7825733d209326b0d5cce57793e216cae4a2dc77ff635f99b66ce6e01b005f"}
Jul 30 17:31:37.570706 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:37.570622    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-4glj8" event={"ID":"d6a88b6a-7228-455f-aa94-57feda298014","Type":"ContainerStarted","Data":"033ef33dd4bca41f75d608cdb047936e32f7b5c155e76f3d567392cbbf3debc0"}
Jul 30 17:31:37.570935 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:37.570917    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-4glj8"
Jul 30 17:31:37.571032 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:37.570965    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-4glj8"
Jul 30 17:31:37.572460 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:37.572435    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-4glj8" podUID="d6a88b6a-7228-455f-aa94-57feda298014" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 30 17:31:37.590739 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:37.590689    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-jkpgt" podStartSLOduration=2.647132361 podStartE2EDuration="6.590676611s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.94354425s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:31:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:31:37.588438226 +0000 UTC m=+1872.075805627" watchObservedRunningTime="2026-07-30 17:31:37.590676611 +0000 UTC m=+1872.078044013"
Jul 30 17:31:37.608514 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:37.608450    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-4glj8" podStartSLOduration=2.729505387 podStartE2EDuration="6.608436978s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.878931591s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:31:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:31:37.606595325 +0000 UTC m=+1872.093962727" watchObservedRunningTime="2026-07-30 17:31:37.608436978 +0000 UTC m=+1872.095804382"
Jul 30 17:31:38.574919 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:38.574881    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-4glj8" podUID="d6a88b6a-7228-455f-aa94-57feda298014" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 30 17:31:38.575395 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:38.574939    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-jkpgt"
Jul 30 17:31:38.576389 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:38.576366    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-jkpgt" podUID="0ed95623-8c1a-430c-bcae-6ac695c65e17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 30 17:31:39.580098 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:39.580053    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-jkpgt" podUID="0ed95623-8c1a-430c-bcae-6ac695c65e17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 30 17:31:43.579187 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:43.579157    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-4glj8"
Jul 30 17:31:43.579649 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:43.579622    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-4glj8" podUID="d6a88b6a-7228-455f-aa94-57feda298014" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 30 17:31:44.584660 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:44.584632    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-jkpgt"
Jul 30 17:31:44.585146 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:44.585122    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-jkpgt" podUID="0ed95623-8c1a-430c-bcae-6ac695c65e17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 30 17:31:53.580325 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:53.580236    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-4glj8" podUID="d6a88b6a-7228-455f-aa94-57feda298014" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 30 17:31:54.585646 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:31:54.585600    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-jkpgt" podUID="0ed95623-8c1a-430c-bcae-6ac695c65e17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 30 17:32:03.579899 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:32:03.579860    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-4glj8" podUID="d6a88b6a-7228-455f-aa94-57feda298014" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 30 17:32:04.585103 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:32:04.585059    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-jkpgt" podUID="0ed95623-8c1a-430c-bcae-6ac695c65e17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 30 17:32:13.580180 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:32:13.580142    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-4glj8" podUID="d6a88b6a-7228-455f-aa94-57feda298014" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 30 17:32:14.585251 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:32:14.585211    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-jkpgt" podUID="0ed95623-8c1a-430c-bcae-6ac695c65e17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 30 17:32:23.580187 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:32:23.580147    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-4glj8" podUID="d6a88b6a-7228-455f-aa94-57feda298014" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 30 17:32:24.585121 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:32:24.585079    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-jkpgt" podUID="0ed95623-8c1a-430c-bcae-6ac695c65e17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 30 17:32:33.580050 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:32:33.579987    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-4glj8" podUID="d6a88b6a-7228-455f-aa94-57feda298014" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 30 17:32:34.585542 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:32:34.585501    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-jkpgt" podUID="0ed95623-8c1a-430c-bcae-6ac695c65e17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 30 17:32:43.580001 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:32:43.579952    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-4glj8" podUID="d6a88b6a-7228-455f-aa94-57feda298014" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 30 17:32:44.585957 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:32:44.585912    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-jkpgt" podUID="0ed95623-8c1a-430c-bcae-6ac695c65e17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 30 17:32:46.063107 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:32:46.063064    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-4glj8" podUID="d6a88b6a-7228-455f-aa94-57feda298014" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 30 17:32:54.586791 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:32:54.586763    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-jkpgt"
Jul 30 17:32:56.064442 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:32:56.064414    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-4glj8"
Jul 30 17:33:01.935379 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:33:01.935345    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-jkpgt"]
Jul 30 17:33:01.935898 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:33:01.935669    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-jkpgt" podUID="0ed95623-8c1a-430c-bcae-6ac695c65e17" containerName="kube-rbac-proxy" containerID="cri-o://e8bce075360f1a85d35ebd4e48652ab97ede242753dcede2c099d3bcafc32ad5" gracePeriod=30
Jul 30 17:33:01.935898 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:33:01.935651    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-jkpgt" podUID="0ed95623-8c1a-430c-bcae-6ac695c65e17" containerName="kserve-container" containerID="cri-o://b8ccfcd86756e016b5010cc1add1b927ee9e0216dc588a5ed1c00ee51e315483" gracePeriod=30
Jul 30 17:33:01.958199 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:33:01.958169    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:33:02.971228 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:33:02.971194    2555 generic.go:356] "Generic (PLEG): container finished" podID="0ed95623-8c1a-430c-bcae-6ac695c65e17" containerID="e8bce075360f1a85d35ebd4e48652ab97ede242753dcede2c099d3bcafc32ad5" exitCode=2
Jul 30 17:33:02.971611 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:33:02.971262    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-jkpgt" event={"ID":"0ed95623-8c1a-430c-bcae-6ac695c65e17","Type":"ContainerDied","Data":"e8bce075360f1a85d35ebd4e48652ab97ede242753dcede2c099d3bcafc32ad5"}
Jul 30 17:33:04.580541 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:33:04.580493    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-jkpgt" podUID="0ed95623-8c1a-430c-bcae-6ac695c65e17" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.38:8643/healthz\": dial tcp 10.133.0.38:8643: connect: connection refused"
Jul 30 17:33:04.585378 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:33:04.585352    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-jkpgt" podUID="0ed95623-8c1a-430c-bcae-6ac695c65e17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 30 17:33:06.689746 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:33:06.689722    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-jkpgt"
Jul 30 17:33:06.817317 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:33:06.817284    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0ed95623-8c1a-430c-bcae-6ac695c65e17-kserve-provision-location\") pod \"0ed95623-8c1a-430c-bcae-6ac695c65e17\" (UID: \"0ed95623-8c1a-430c-bcae-6ac695c65e17\") "
Jul 30 17:33:06.817490 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:33:06.817356    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fz9dr\" (UniqueName: \"kubernetes.io/projected/0ed95623-8c1a-430c-bcae-6ac695c65e17-kube-api-access-fz9dr\") pod \"0ed95623-8c1a-430c-bcae-6ac695c65e17\" (UID: \"0ed95623-8c1a-430c-bcae-6ac695c65e17\") "
Jul 30 17:33:06.817490 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:33:06.817400    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0ed95623-8c1a-430c-bcae-6ac695c65e17-kube-rbac-proxy-sar-config\") pod \"0ed95623-8c1a-430c-bcae-6ac695c65e17\" (UID: \"0ed95623-8c1a-430c-bcae-6ac695c65e17\") "
Jul 30 17:33:06.817490 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:33:06.817431    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0ed95623-8c1a-430c-bcae-6ac695c65e17-proxy-tls\") pod \"0ed95623-8c1a-430c-bcae-6ac695c65e17\" (UID: \"0ed95623-8c1a-430c-bcae-6ac695c65e17\") "
Jul 30 17:33:06.817490 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:33:06.817472    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0ed95623-8c1a-430c-bcae-6ac695c65e17-cabundle-cert\") pod \"0ed95623-8c1a-430c-bcae-6ac695c65e17\" (UID: \"0ed95623-8c1a-430c-bcae-6ac695c65e17\") "
Jul 30 17:33:06.817728 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:33:06.817661    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ed95623-8c1a-430c-bcae-6ac695c65e17-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0ed95623-8c1a-430c-bcae-6ac695c65e17" (UID: "0ed95623-8c1a-430c-bcae-6ac695c65e17"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 17:33:06.817785 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:33:06.817768    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0ed95623-8c1a-430c-bcae-6ac695c65e17-kserve-provision-location\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:33:06.817875 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:33:06.817848    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ed95623-8c1a-430c-bcae-6ac695c65e17-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0ed95623-8c1a-430c-bcae-6ac695c65e17" (UID: "0ed95623-8c1a-430c-bcae-6ac695c65e17"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:33:06.817933 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:33:06.817906    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ed95623-8c1a-430c-bcae-6ac695c65e17-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0ed95623-8c1a-430c-bcae-6ac695c65e17" (UID: "0ed95623-8c1a-430c-bcae-6ac695c65e17"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:33:06.819470 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:33:06.819449    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ed95623-8c1a-430c-bcae-6ac695c65e17-kube-api-access-fz9dr" (OuterVolumeSpecName: "kube-api-access-fz9dr") pod "0ed95623-8c1a-430c-bcae-6ac695c65e17" (UID: "0ed95623-8c1a-430c-bcae-6ac695c65e17"). InnerVolumeSpecName "kube-api-access-fz9dr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 17:33:06.819470 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:33:06.819462    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ed95623-8c1a-430c-bcae-6ac695c65e17-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0ed95623-8c1a-430c-bcae-6ac695c65e17" (UID: "0ed95623-8c1a-430c-bcae-6ac695c65e17"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:33:06.918514 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:33:06.918466    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fz9dr\" (UniqueName: \"kubernetes.io/projected/0ed95623-8c1a-430c-bcae-6ac695c65e17-kube-api-access-fz9dr\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:33:06.918514 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:33:06.918508    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0ed95623-8c1a-430c-bcae-6ac695c65e17-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:33:06.918514 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:33:06.918520    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0ed95623-8c1a-430c-bcae-6ac695c65e17-proxy-tls\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:33:06.918514 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:33:06.918533    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0ed95623-8c1a-430c-bcae-6ac695c65e17-cabundle-cert\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:33:06.992377 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:33:06.992345    2555 generic.go:356] "Generic (PLEG): container finished" podID="0ed95623-8c1a-430c-bcae-6ac695c65e17" containerID="b8ccfcd86756e016b5010cc1add1b927ee9e0216dc588a5ed1c00ee51e315483" exitCode=0
Jul 30 17:33:06.992564 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:33:06.992406    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-jkpgt"
Jul 30 17:33:06.992564 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:33:06.992420    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-jkpgt" event={"ID":"0ed95623-8c1a-430c-bcae-6ac695c65e17","Type":"ContainerDied","Data":"b8ccfcd86756e016b5010cc1add1b927ee9e0216dc588a5ed1c00ee51e315483"}
Jul 30 17:33:06.992564 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:33:06.992477    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-jkpgt" event={"ID":"0ed95623-8c1a-430c-bcae-6ac695c65e17","Type":"ContainerDied","Data":"94249d8e9620df27c25d7bdd906b79279871d38db2366fdfd238375574997a3f"}
Jul 30 17:33:06.992564 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:33:06.992504    2555 scope.go:117] "RemoveContainer" containerID="e8bce075360f1a85d35ebd4e48652ab97ede242753dcede2c099d3bcafc32ad5"
Jul 30 17:33:07.001512 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:33:07.001497    2555 scope.go:117] "RemoveContainer" containerID="b8ccfcd86756e016b5010cc1add1b927ee9e0216dc588a5ed1c00ee51e315483"
Jul 30 17:33:07.008986 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:33:07.008971    2555 scope.go:117] "RemoveContainer" containerID="fc1f7cf678e673a718614688d2ab015cb436fb0043538497bf18e67fd3bbd305"
Jul 30 17:33:07.017108 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:33:07.017081    2555 scope.go:117] "RemoveContainer" containerID="e8bce075360f1a85d35ebd4e48652ab97ede242753dcede2c099d3bcafc32ad5"
Jul 30 17:33:07.017391 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:33:07.017374    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8bce075360f1a85d35ebd4e48652ab97ede242753dcede2c099d3bcafc32ad5\": container with ID starting with e8bce075360f1a85d35ebd4e48652ab97ede242753dcede2c099d3bcafc32ad5 not found: ID does not exist" containerID="e8bce075360f1a85d35ebd4e48652ab97ede242753dcede2c099d3bcafc32ad5"
Jul 30 17:33:07.017468 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:33:07.017400    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8bce075360f1a85d35ebd4e48652ab97ede242753dcede2c099d3bcafc32ad5"} err="failed to get container status \"e8bce075360f1a85d35ebd4e48652ab97ede242753dcede2c099d3bcafc32ad5\": rpc error: code = NotFound desc = could not find container \"e8bce075360f1a85d35ebd4e48652ab97ede242753dcede2c099d3bcafc32ad5\": container with ID starting with e8bce075360f1a85d35ebd4e48652ab97ede242753dcede2c099d3bcafc32ad5 not found: ID does not exist"
Jul 30 17:33:07.017468 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:33:07.017418    2555 scope.go:117] "RemoveContainer" containerID="b8ccfcd86756e016b5010cc1add1b927ee9e0216dc588a5ed1c00ee51e315483"
Jul 30 17:33:07.017667 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:33:07.017648    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8ccfcd86756e016b5010cc1add1b927ee9e0216dc588a5ed1c00ee51e315483\": container with ID starting with b8ccfcd86756e016b5010cc1add1b927ee9e0216dc588a5ed1c00ee51e315483 not found: ID does not exist" containerID="b8ccfcd86756e016b5010cc1add1b927ee9e0216dc588a5ed1c00ee51e315483"
Jul 30 17:33:07.017712 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:33:07.017677    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8ccfcd86756e016b5010cc1add1b927ee9e0216dc588a5ed1c00ee51e315483"} err="failed to get container status \"b8ccfcd86756e016b5010cc1add1b927ee9e0216dc588a5ed1c00ee51e315483\": rpc error: code = NotFound desc = could not find container \"b8ccfcd86756e016b5010cc1add1b927ee9e0216dc588a5ed1c00ee51e315483\": container with ID starting with b8ccfcd86756e016b5010cc1add1b927ee9e0216dc588a5ed1c00ee51e315483 not found: ID does not exist"
Jul 30 17:33:07.017712 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:33:07.017691    2555 scope.go:117] "RemoveContainer" containerID="fc1f7cf678e673a718614688d2ab015cb436fb0043538497bf18e67fd3bbd305"
Jul 30 17:33:07.017795 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:33:07.017732    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-jkpgt"]
Jul 30 17:33:07.017918 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:33:07.017902    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc1f7cf678e673a718614688d2ab015cb436fb0043538497bf18e67fd3bbd305\": container with ID starting with fc1f7cf678e673a718614688d2ab015cb436fb0043538497bf18e67fd3bbd305 not found: ID does not exist" containerID="fc1f7cf678e673a718614688d2ab015cb436fb0043538497bf18e67fd3bbd305"
Jul 30 17:33:07.017961 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:33:07.017922    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc1f7cf678e673a718614688d2ab015cb436fb0043538497bf18e67fd3bbd305"} err="failed to get container status \"fc1f7cf678e673a718614688d2ab015cb436fb0043538497bf18e67fd3bbd305\": rpc error: code = NotFound desc = could not find container \"fc1f7cf678e673a718614688d2ab015cb436fb0043538497bf18e67fd3bbd305\": container with ID starting with fc1f7cf678e673a718614688d2ab015cb436fb0043538497bf18e67fd3bbd305 not found: ID does not exist"
Jul 30 17:33:07.021972 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:33:07.021951    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-578c74fc8-jkpgt"]
Jul 30 17:33:08.060662 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:33:08.060627    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ed95623-8c1a-430c-bcae-6ac695c65e17" path="/var/lib/kubelet/pods/0ed95623-8c1a-430c-bcae-6ac695c65e17/volumes"
Jul 30 17:33:09.558751 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:33:09.558719    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:34:32.170122 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:32.170074    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-4glj8"]
Jul 30 17:34:32.170747 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:32.170403    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-4glj8" podUID="d6a88b6a-7228-455f-aa94-57feda298014" containerName="kserve-container" containerID="cri-o://033ef33dd4bca41f75d608cdb047936e32f7b5c155e76f3d567392cbbf3debc0" gracePeriod=30
Jul 30 17:34:32.170819 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:32.170705    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-4glj8" podUID="d6a88b6a-7228-455f-aa94-57feda298014" containerName="kube-rbac-proxy" containerID="cri-o://9a7825733d209326b0d5cce57793e216cae4a2dc77ff635f99b66ce6e01b005f" gracePeriod=30
Jul 30 17:34:32.198924 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:32.198895    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:34:32.385211 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:32.385176    2555 generic.go:356] "Generic (PLEG): container finished" podID="d6a88b6a-7228-455f-aa94-57feda298014" containerID="9a7825733d209326b0d5cce57793e216cae4a2dc77ff635f99b66ce6e01b005f" exitCode=2
Jul 30 17:34:32.385426 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:32.385256    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-4glj8" event={"ID":"d6a88b6a-7228-455f-aa94-57feda298014","Type":"ContainerDied","Data":"9a7825733d209326b0d5cce57793e216cae4a2dc77ff635f99b66ce6e01b005f"}
Jul 30 17:34:33.575739 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:33.575695    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-4glj8" podUID="d6a88b6a-7228-455f-aa94-57feda298014" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
Jul 30 17:34:36.063639 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:36.063602    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-4glj8" podUID="d6a88b6a-7228-455f-aa94-57feda298014" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 30 17:34:36.926000 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:36.925978    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-4glj8"
Jul 30 17:34:36.977551 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:36.977476    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6a88b6a-7228-455f-aa94-57feda298014-kube-rbac-proxy-sar-config\") pod \"d6a88b6a-7228-455f-aa94-57feda298014\" (UID: \"d6a88b6a-7228-455f-aa94-57feda298014\") "
Jul 30 17:34:36.977551 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:36.977530    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6a88b6a-7228-455f-aa94-57feda298014-kserve-provision-location\") pod \"d6a88b6a-7228-455f-aa94-57feda298014\" (UID: \"d6a88b6a-7228-455f-aa94-57feda298014\") "
Jul 30 17:34:36.977777 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:36.977564    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rfq8z\" (UniqueName: \"kubernetes.io/projected/d6a88b6a-7228-455f-aa94-57feda298014-kube-api-access-rfq8z\") pod \"d6a88b6a-7228-455f-aa94-57feda298014\" (UID: \"d6a88b6a-7228-455f-aa94-57feda298014\") "
Jul 30 17:34:36.977777 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:36.977596    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6a88b6a-7228-455f-aa94-57feda298014-proxy-tls\") pod \"d6a88b6a-7228-455f-aa94-57feda298014\" (UID: \"d6a88b6a-7228-455f-aa94-57feda298014\") "
Jul 30 17:34:36.977777 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:36.977658    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6a88b6a-7228-455f-aa94-57feda298014-cabundle-cert\") pod \"d6a88b6a-7228-455f-aa94-57feda298014\" (UID: \"d6a88b6a-7228-455f-aa94-57feda298014\") "
Jul 30 17:34:36.977942 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:36.977824    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6a88b6a-7228-455f-aa94-57feda298014-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d6a88b6a-7228-455f-aa94-57feda298014" (UID: "d6a88b6a-7228-455f-aa94-57feda298014"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:34:36.977942 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:36.977888    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6a88b6a-7228-455f-aa94-57feda298014-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d6a88b6a-7228-455f-aa94-57feda298014" (UID: "d6a88b6a-7228-455f-aa94-57feda298014"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 17:34:36.978103 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:36.978087    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6a88b6a-7228-455f-aa94-57feda298014-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d6a88b6a-7228-455f-aa94-57feda298014" (UID: "d6a88b6a-7228-455f-aa94-57feda298014"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:34:36.979657 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:36.979641    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6a88b6a-7228-455f-aa94-57feda298014-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d6a88b6a-7228-455f-aa94-57feda298014" (UID: "d6a88b6a-7228-455f-aa94-57feda298014"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:34:36.979810 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:36.979790    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6a88b6a-7228-455f-aa94-57feda298014-kube-api-access-rfq8z" (OuterVolumeSpecName: "kube-api-access-rfq8z") pod "d6a88b6a-7228-455f-aa94-57feda298014" (UID: "d6a88b6a-7228-455f-aa94-57feda298014"). InnerVolumeSpecName "kube-api-access-rfq8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 17:34:37.078545 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.078493    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfq8z\" (UniqueName: \"kubernetes.io/projected/d6a88b6a-7228-455f-aa94-57feda298014-kube-api-access-rfq8z\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:34:37.078545 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.078539    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6a88b6a-7228-455f-aa94-57feda298014-proxy-tls\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:34:37.078545 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.078550    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6a88b6a-7228-455f-aa94-57feda298014-cabundle-cert\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:34:37.078545 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.078559    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6a88b6a-7228-455f-aa94-57feda298014-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:34:37.079074 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.078568    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6a88b6a-7228-455f-aa94-57feda298014-kserve-provision-location\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:34:37.241627 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.241539    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:34:37.293162 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.293130    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-hntx2"]
Jul 30 17:34:37.293494 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.293481    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6a88b6a-7228-455f-aa94-57feda298014" containerName="storage-initializer"
Jul 30 17:34:37.293551 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.293496    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6a88b6a-7228-455f-aa94-57feda298014" containerName="storage-initializer"
Jul 30 17:34:37.293551 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.293504    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6a88b6a-7228-455f-aa94-57feda298014" containerName="kube-rbac-proxy"
Jul 30 17:34:37.293551 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.293510    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6a88b6a-7228-455f-aa94-57feda298014" containerName="kube-rbac-proxy"
Jul 30 17:34:37.293551 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.293523    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ed95623-8c1a-430c-bcae-6ac695c65e17" containerName="storage-initializer"
Jul 30 17:34:37.293551 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.293528    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ed95623-8c1a-430c-bcae-6ac695c65e17" containerName="storage-initializer"
Jul 30 17:34:37.293551 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.293538    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6a88b6a-7228-455f-aa94-57feda298014" containerName="kserve-container"
Jul 30 17:34:37.293551 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.293543    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6a88b6a-7228-455f-aa94-57feda298014" containerName="kserve-container"
Jul 30 17:34:37.293759 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.293556    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ed95623-8c1a-430c-bcae-6ac695c65e17" containerName="kserve-container"
Jul 30 17:34:37.293759 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.293562    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ed95623-8c1a-430c-bcae-6ac695c65e17" containerName="kserve-container"
Jul 30 17:34:37.293759 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.293572    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ed95623-8c1a-430c-bcae-6ac695c65e17" containerName="kube-rbac-proxy"
Jul 30 17:34:37.293759 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.293577    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ed95623-8c1a-430c-bcae-6ac695c65e17" containerName="kube-rbac-proxy"
Jul 30 17:34:37.293759 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.293629    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ed95623-8c1a-430c-bcae-6ac695c65e17" containerName="kube-rbac-proxy"
Jul 30 17:34:37.293759 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.293636    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6a88b6a-7228-455f-aa94-57feda298014" containerName="kube-rbac-proxy"
Jul 30 17:34:37.293759 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.293645    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ed95623-8c1a-430c-bcae-6ac695c65e17" containerName="kserve-container"
Jul 30 17:34:37.293759 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.293651    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6a88b6a-7228-455f-aa94-57feda298014" containerName="kserve-container"
Jul 30 17:34:37.297064 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.297043    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-hntx2"
Jul 30 17:34:37.299304 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.299276    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-784v2"]
Jul 30 17:34:37.299884 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.299864    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Jul 30 17:34:37.299976 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.299870    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Jul 30 17:34:37.303620 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.303592    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-784v2"
Jul 30 17:34:37.307078 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.307061    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Jul 30 17:34:37.311425 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.311403    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-hntx2"]
Jul 30 17:34:37.314638 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.314585    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-784v2"]
Jul 30 17:34:37.322039 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.322019    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:34:37.381780 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.381744    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/084bf132-1604-41f5-9b85-dcff676b84e9-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-9b8c9778-784v2\" (UID: \"084bf132-1604-41f5-9b85-dcff676b84e9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-784v2"
Jul 30 17:34:37.381961 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.381848    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/af5aecee-8785-40df-a16d-5361f4aa8a9e-cabundle-cert\") pod \"isvc-canary-stop-predictor-77665d5899-hntx2\" (UID: \"af5aecee-8785-40df-a16d-5361f4aa8a9e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-hntx2"
Jul 30 17:34:37.381961 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.381882    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/084bf132-1604-41f5-9b85-dcff676b84e9-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-9b8c9778-784v2\" (UID: \"084bf132-1604-41f5-9b85-dcff676b84e9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-784v2"
Jul 30 17:34:37.381961 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.381935    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/084bf132-1604-41f5-9b85-dcff676b84e9-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-9b8c9778-784v2\" (UID: \"084bf132-1604-41f5-9b85-dcff676b84e9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-784v2"
Jul 30 17:34:37.382111 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.382045    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9znss\" (UniqueName: \"kubernetes.io/projected/084bf132-1604-41f5-9b85-dcff676b84e9-kube-api-access-9znss\") pod \"isvc-canary-stop-v2-predictor-9b8c9778-784v2\" (UID: \"084bf132-1604-41f5-9b85-dcff676b84e9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-784v2"
Jul 30 17:34:37.382111 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.382074    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/af5aecee-8785-40df-a16d-5361f4aa8a9e-kserve-provision-location\") pod \"isvc-canary-stop-predictor-77665d5899-hntx2\" (UID: \"af5aecee-8785-40df-a16d-5361f4aa8a9e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-hntx2"
Jul 30 17:34:37.382185 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.382115    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/af5aecee-8785-40df-a16d-5361f4aa8a9e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-77665d5899-hntx2\" (UID: \"af5aecee-8785-40df-a16d-5361f4aa8a9e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-hntx2"
Jul 30 17:34:37.382185 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.382147    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d88qs\" (UniqueName: \"kubernetes.io/projected/af5aecee-8785-40df-a16d-5361f4aa8a9e-kube-api-access-d88qs\") pod \"isvc-canary-stop-predictor-77665d5899-hntx2\" (UID: \"af5aecee-8785-40df-a16d-5361f4aa8a9e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-hntx2"
Jul 30 17:34:37.382185 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.382170    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/084bf132-1604-41f5-9b85-dcff676b84e9-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-9b8c9778-784v2\" (UID: \"084bf132-1604-41f5-9b85-dcff676b84e9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-784v2"
Jul 30 17:34:37.382332 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.382228    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af5aecee-8785-40df-a16d-5361f4aa8a9e-proxy-tls\") pod \"isvc-canary-stop-predictor-77665d5899-hntx2\" (UID: \"af5aecee-8785-40df-a16d-5361f4aa8a9e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-hntx2"
Jul 30 17:34:37.405705 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.405674    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:34:37.410355 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.410331    2555 generic.go:356] "Generic (PLEG): container finished" podID="d6a88b6a-7228-455f-aa94-57feda298014" containerID="033ef33dd4bca41f75d608cdb047936e32f7b5c155e76f3d567392cbbf3debc0" exitCode=0
Jul 30 17:34:37.410475 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.410399    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-4glj8"
Jul 30 17:34:37.410475 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.410413    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-4glj8" event={"ID":"d6a88b6a-7228-455f-aa94-57feda298014","Type":"ContainerDied","Data":"033ef33dd4bca41f75d608cdb047936e32f7b5c155e76f3d567392cbbf3debc0"}
Jul 30 17:34:37.410475 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.410474    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-4glj8" event={"ID":"d6a88b6a-7228-455f-aa94-57feda298014","Type":"ContainerDied","Data":"1d62b971922d41286044b0f29435cc5494c27e4e6a06501468eb053c0e434be7"}
Jul 30 17:34:37.410577 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.410501    2555 scope.go:117] "RemoveContainer" containerID="9a7825733d209326b0d5cce57793e216cae4a2dc77ff635f99b66ce6e01b005f"
Jul 30 17:34:37.419381 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.419367    2555 scope.go:117] "RemoveContainer" containerID="033ef33dd4bca41f75d608cdb047936e32f7b5c155e76f3d567392cbbf3debc0"
Jul 30 17:34:37.426926 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.426908    2555 scope.go:117] "RemoveContainer" containerID="ff84720bdfa54241cfc34b54326dc3fc7af578aac791a3609ec5587a049fd26b"
Jul 30 17:34:37.434757 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.434741    2555 scope.go:117] "RemoveContainer" containerID="9a7825733d209326b0d5cce57793e216cae4a2dc77ff635f99b66ce6e01b005f"
Jul 30 17:34:37.435026 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:34:37.434983    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a7825733d209326b0d5cce57793e216cae4a2dc77ff635f99b66ce6e01b005f\": container with ID starting with 9a7825733d209326b0d5cce57793e216cae4a2dc77ff635f99b66ce6e01b005f not found: ID does not exist" containerID="9a7825733d209326b0d5cce57793e216cae4a2dc77ff635f99b66ce6e01b005f"
Jul 30 17:34:37.435101 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.435033    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a7825733d209326b0d5cce57793e216cae4a2dc77ff635f99b66ce6e01b005f"} err="failed to get container status \"9a7825733d209326b0d5cce57793e216cae4a2dc77ff635f99b66ce6e01b005f\": rpc error: code = NotFound desc = could not find container \"9a7825733d209326b0d5cce57793e216cae4a2dc77ff635f99b66ce6e01b005f\": container with ID starting with 9a7825733d209326b0d5cce57793e216cae4a2dc77ff635f99b66ce6e01b005f not found: ID does not exist"
Jul 30 17:34:37.435101 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.435049    2555 scope.go:117] "RemoveContainer" containerID="033ef33dd4bca41f75d608cdb047936e32f7b5c155e76f3d567392cbbf3debc0"
Jul 30 17:34:37.435308 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:34:37.435293    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"033ef33dd4bca41f75d608cdb047936e32f7b5c155e76f3d567392cbbf3debc0\": container with ID starting with 033ef33dd4bca41f75d608cdb047936e32f7b5c155e76f3d567392cbbf3debc0 not found: ID does not exist" containerID="033ef33dd4bca41f75d608cdb047936e32f7b5c155e76f3d567392cbbf3debc0"
Jul 30 17:34:37.435355 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.435310    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"033ef33dd4bca41f75d608cdb047936e32f7b5c155e76f3d567392cbbf3debc0"} err="failed to get container status \"033ef33dd4bca41f75d608cdb047936e32f7b5c155e76f3d567392cbbf3debc0\": rpc error: code = NotFound desc = could not find container \"033ef33dd4bca41f75d608cdb047936e32f7b5c155e76f3d567392cbbf3debc0\": container with ID starting with 033ef33dd4bca41f75d608cdb047936e32f7b5c155e76f3d567392cbbf3debc0 not found: ID does not exist"
Jul 30 17:34:37.435355 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.435321    2555 scope.go:117] "RemoveContainer" containerID="ff84720bdfa54241cfc34b54326dc3fc7af578aac791a3609ec5587a049fd26b"
Jul 30 17:34:37.435540 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:34:37.435499    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff84720bdfa54241cfc34b54326dc3fc7af578aac791a3609ec5587a049fd26b\": container with ID starting with ff84720bdfa54241cfc34b54326dc3fc7af578aac791a3609ec5587a049fd26b not found: ID does not exist" containerID="ff84720bdfa54241cfc34b54326dc3fc7af578aac791a3609ec5587a049fd26b"
Jul 30 17:34:37.435540 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.435519    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff84720bdfa54241cfc34b54326dc3fc7af578aac791a3609ec5587a049fd26b"} err="failed to get container status \"ff84720bdfa54241cfc34b54326dc3fc7af578aac791a3609ec5587a049fd26b\": rpc error: code = NotFound desc = could not find container \"ff84720bdfa54241cfc34b54326dc3fc7af578aac791a3609ec5587a049fd26b\": container with ID starting with ff84720bdfa54241cfc34b54326dc3fc7af578aac791a3609ec5587a049fd26b not found: ID does not exist"
Jul 30 17:34:37.436490 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.436470    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-4glj8"]
Jul 30 17:34:37.442503 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.442482    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-4glj8"]
Jul 30 17:34:37.483167 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.483137    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/af5aecee-8785-40df-a16d-5361f4aa8a9e-cabundle-cert\") pod \"isvc-canary-stop-predictor-77665d5899-hntx2\" (UID: \"af5aecee-8785-40df-a16d-5361f4aa8a9e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-hntx2"
Jul 30 17:34:37.483306 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.483173    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/084bf132-1604-41f5-9b85-dcff676b84e9-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-9b8c9778-784v2\" (UID: \"084bf132-1604-41f5-9b85-dcff676b84e9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-784v2"
Jul 30 17:34:37.483306 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.483194    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/084bf132-1604-41f5-9b85-dcff676b84e9-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-9b8c9778-784v2\" (UID: \"084bf132-1604-41f5-9b85-dcff676b84e9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-784v2"
Jul 30 17:34:37.483306 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.483242    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9znss\" (UniqueName: \"kubernetes.io/projected/084bf132-1604-41f5-9b85-dcff676b84e9-kube-api-access-9znss\") pod \"isvc-canary-stop-v2-predictor-9b8c9778-784v2\" (UID: \"084bf132-1604-41f5-9b85-dcff676b84e9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-784v2"
Jul 30 17:34:37.483306 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.483263    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/af5aecee-8785-40df-a16d-5361f4aa8a9e-kserve-provision-location\") pod \"isvc-canary-stop-predictor-77665d5899-hntx2\" (UID: \"af5aecee-8785-40df-a16d-5361f4aa8a9e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-hntx2"
Jul 30 17:34:37.483306 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.483293    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/af5aecee-8785-40df-a16d-5361f4aa8a9e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-77665d5899-hntx2\" (UID: \"af5aecee-8785-40df-a16d-5361f4aa8a9e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-hntx2"
Jul 30 17:34:37.483631 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.483326    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d88qs\" (UniqueName: \"kubernetes.io/projected/af5aecee-8785-40df-a16d-5361f4aa8a9e-kube-api-access-d88qs\") pod \"isvc-canary-stop-predictor-77665d5899-hntx2\" (UID: \"af5aecee-8785-40df-a16d-5361f4aa8a9e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-hntx2"
Jul 30 17:34:37.483631 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.483349    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/084bf132-1604-41f5-9b85-dcff676b84e9-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-9b8c9778-784v2\" (UID: \"084bf132-1604-41f5-9b85-dcff676b84e9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-784v2"
Jul 30 17:34:37.483631 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.483376    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af5aecee-8785-40df-a16d-5361f4aa8a9e-proxy-tls\") pod \"isvc-canary-stop-predictor-77665d5899-hntx2\" (UID: \"af5aecee-8785-40df-a16d-5361f4aa8a9e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-hntx2"
Jul 30 17:34:37.483631 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.483407    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/084bf132-1604-41f5-9b85-dcff676b84e9-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-9b8c9778-784v2\" (UID: \"084bf132-1604-41f5-9b85-dcff676b84e9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-784v2"
Jul 30 17:34:37.483841 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.483662    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/084bf132-1604-41f5-9b85-dcff676b84e9-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-9b8c9778-784v2\" (UID: \"084bf132-1604-41f5-9b85-dcff676b84e9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-784v2"
Jul 30 17:34:37.483841 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:34:37.483752    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-predictor-serving-cert: secret "isvc-canary-stop-predictor-serving-cert" not found
Jul 30 17:34:37.483841 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:34:37.483826    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af5aecee-8785-40df-a16d-5361f4aa8a9e-proxy-tls podName:af5aecee-8785-40df-a16d-5361f4aa8a9e nodeName:}" failed. No retries permitted until 2026-07-30 17:34:37.983806535 +0000 UTC m=+2052.471173916 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/af5aecee-8785-40df-a16d-5361f4aa8a9e-proxy-tls") pod "isvc-canary-stop-predictor-77665d5899-hntx2" (UID: "af5aecee-8785-40df-a16d-5361f4aa8a9e") : secret "isvc-canary-stop-predictor-serving-cert" not found
Jul 30 17:34:37.484002 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.483982    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/084bf132-1604-41f5-9b85-dcff676b84e9-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-9b8c9778-784v2\" (UID: \"084bf132-1604-41f5-9b85-dcff676b84e9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-784v2"
Jul 30 17:34:37.484002 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.483997    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/af5aecee-8785-40df-a16d-5361f4aa8a9e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-77665d5899-hntx2\" (UID: \"af5aecee-8785-40df-a16d-5361f4aa8a9e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-hntx2"
Jul 30 17:34:37.484150 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.483982    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/af5aecee-8785-40df-a16d-5361f4aa8a9e-cabundle-cert\") pod \"isvc-canary-stop-predictor-77665d5899-hntx2\" (UID: \"af5aecee-8785-40df-a16d-5361f4aa8a9e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-hntx2"
Jul 30 17:34:37.484150 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.484101    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/af5aecee-8785-40df-a16d-5361f4aa8a9e-kserve-provision-location\") pod \"isvc-canary-stop-predictor-77665d5899-hntx2\" (UID: \"af5aecee-8785-40df-a16d-5361f4aa8a9e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-hntx2"
Jul 30 17:34:37.484308 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.484287    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/084bf132-1604-41f5-9b85-dcff676b84e9-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-9b8c9778-784v2\" (UID: \"084bf132-1604-41f5-9b85-dcff676b84e9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-784v2"
Jul 30 17:34:37.485978 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.485961    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/084bf132-1604-41f5-9b85-dcff676b84e9-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-9b8c9778-784v2\" (UID: \"084bf132-1604-41f5-9b85-dcff676b84e9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-784v2"
Jul 30 17:34:37.495457 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.495393    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d88qs\" (UniqueName: \"kubernetes.io/projected/af5aecee-8785-40df-a16d-5361f4aa8a9e-kube-api-access-d88qs\") pod \"isvc-canary-stop-predictor-77665d5899-hntx2\" (UID: \"af5aecee-8785-40df-a16d-5361f4aa8a9e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-hntx2"
Jul 30 17:34:37.495576 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.495514    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9znss\" (UniqueName: \"kubernetes.io/projected/084bf132-1604-41f5-9b85-dcff676b84e9-kube-api-access-9znss\") pod \"isvc-canary-stop-v2-predictor-9b8c9778-784v2\" (UID: \"084bf132-1604-41f5-9b85-dcff676b84e9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-784v2"
Jul 30 17:34:37.554168 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.554139    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:34:37.618402 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.618362    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-784v2"
Jul 30 17:34:37.755234 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.755205    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-784v2"]
Jul 30 17:34:37.757179 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:34:37.757151    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod084bf132_1604_41f5_9b85_dcff676b84e9.slice/crio-38616051f852ab62d7a811505656acc8925a4848a97c4238b7286ac03b44c7bf WatchSource:0}: Error finding container 38616051f852ab62d7a811505656acc8925a4848a97c4238b7286ac03b44c7bf: Status 404 returned error can't find the container with id 38616051f852ab62d7a811505656acc8925a4848a97c4238b7286ac03b44c7bf
Jul 30 17:34:37.989311 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.989272    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af5aecee-8785-40df-a16d-5361f4aa8a9e-proxy-tls\") pod \"isvc-canary-stop-predictor-77665d5899-hntx2\" (UID: \"af5aecee-8785-40df-a16d-5361f4aa8a9e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-hntx2"
Jul 30 17:34:37.991773 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:37.991743    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af5aecee-8785-40df-a16d-5361f4aa8a9e-proxy-tls\") pod \"isvc-canary-stop-predictor-77665d5899-hntx2\" (UID: \"af5aecee-8785-40df-a16d-5361f4aa8a9e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-hntx2"
Jul 30 17:34:38.062348 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:38.062316    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6a88b6a-7228-455f-aa94-57feda298014" path="/var/lib/kubelet/pods/d6a88b6a-7228-455f-aa94-57feda298014/volumes"
Jul 30 17:34:38.208773 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:38.208738    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-hntx2"
Jul 30 17:34:38.349175 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:38.349150    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-hntx2"]
Jul 30 17:34:38.351387 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:34:38.351361    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf5aecee_8785_40df_a16d_5361f4aa8a9e.slice/crio-34cb87bfb36e8720a13695f0052f0a3740c48f32e43c737ef6f088e2e87155b3 WatchSource:0}: Error finding container 34cb87bfb36e8720a13695f0052f0a3740c48f32e43c737ef6f088e2e87155b3: Status 404 returned error can't find the container with id 34cb87bfb36e8720a13695f0052f0a3740c48f32e43c737ef6f088e2e87155b3
Jul 30 17:34:38.416588 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:38.416558    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-784v2" event={"ID":"084bf132-1604-41f5-9b85-dcff676b84e9","Type":"ContainerStarted","Data":"ff4c8f6c7d02d1deec8920a80d27b5f0c11debcfdfcfefa45a205b8439f2e73a"}
Jul 30 17:34:38.416725 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:38.416628    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-784v2" event={"ID":"084bf132-1604-41f5-9b85-dcff676b84e9","Type":"ContainerStarted","Data":"38616051f852ab62d7a811505656acc8925a4848a97c4238b7286ac03b44c7bf"}
Jul 30 17:34:38.419160 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:38.419139    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-hntx2" event={"ID":"af5aecee-8785-40df-a16d-5361f4aa8a9e","Type":"ContainerStarted","Data":"34cb87bfb36e8720a13695f0052f0a3740c48f32e43c737ef6f088e2e87155b3"}
Jul 30 17:34:39.426076 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:39.426038    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-hntx2" event={"ID":"af5aecee-8785-40df-a16d-5361f4aa8a9e","Type":"ContainerStarted","Data":"8e8aea58c0e6fc9b42881fc56aed3809047ba859ffef420473856cc00575d3d9"}
Jul 30 17:34:42.444699 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:42.444664    2555 generic.go:356] "Generic (PLEG): container finished" podID="084bf132-1604-41f5-9b85-dcff676b84e9" containerID="ff4c8f6c7d02d1deec8920a80d27b5f0c11debcfdfcfefa45a205b8439f2e73a" exitCode=0
Jul 30 17:34:42.445206 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:42.444744    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-784v2" event={"ID":"084bf132-1604-41f5-9b85-dcff676b84e9","Type":"ContainerDied","Data":"ff4c8f6c7d02d1deec8920a80d27b5f0c11debcfdfcfefa45a205b8439f2e73a"}
Jul 30 17:34:42.446433 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:42.446409    2555 generic.go:356] "Generic (PLEG): container finished" podID="af5aecee-8785-40df-a16d-5361f4aa8a9e" containerID="8e8aea58c0e6fc9b42881fc56aed3809047ba859ffef420473856cc00575d3d9" exitCode=0
Jul 30 17:34:42.446541 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:42.446441    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-hntx2" event={"ID":"af5aecee-8785-40df-a16d-5361f4aa8a9e","Type":"ContainerDied","Data":"8e8aea58c0e6fc9b42881fc56aed3809047ba859ffef420473856cc00575d3d9"}
Jul 30 17:34:43.453460 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:43.453420    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-hntx2" event={"ID":"af5aecee-8785-40df-a16d-5361f4aa8a9e","Type":"ContainerStarted","Data":"dd83b4829088514f025df178d2a99ac1ee828be8b31ef3cc5d6655b9505e15da"}
Jul 30 17:34:43.453908 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:43.453494    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-hntx2" event={"ID":"af5aecee-8785-40df-a16d-5361f4aa8a9e","Type":"ContainerStarted","Data":"12818f4774854176b433704d6be440bfe3e1166deb34f50ff0a89b5610dd1013"}
Jul 30 17:34:43.453908 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:43.453804    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-hntx2"
Jul 30 17:34:43.454045 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:43.453972    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-hntx2"
Jul 30 17:34:43.455675 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:43.455649    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-hntx2" podUID="af5aecee-8785-40df-a16d-5361f4aa8a9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 30 17:34:43.455809 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:43.455699    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-784v2" event={"ID":"084bf132-1604-41f5-9b85-dcff676b84e9","Type":"ContainerStarted","Data":"0ff6c0bc959d629eb6ac36949bfb38ce42f16f5ea7a4361774a9bb334bab2bd7"}
Jul 30 17:34:43.455809 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:43.455739    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-784v2" event={"ID":"084bf132-1604-41f5-9b85-dcff676b84e9","Type":"ContainerStarted","Data":"cdd86790b98f3491684e57a8769c578b2b128005aebdba866247b363f062e6c8"}
Jul 30 17:34:43.455997 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:43.455982    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-784v2"
Jul 30 17:34:43.456073 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:43.456047    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-784v2"
Jul 30 17:34:43.457094 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:43.457072    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-784v2" podUID="084bf132-1604-41f5-9b85-dcff676b84e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 30 17:34:43.480341 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:43.480295    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-hntx2" podStartSLOduration=2.870674916 podStartE2EDuration="6.480280146s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.60960523s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:34:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:34:43.478356789 +0000 UTC m=+2057.965724202" watchObservedRunningTime="2026-07-30 17:34:43.480280146 +0000 UTC m=+2057.967647607"
Jul 30 17:34:43.505494 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:43.505444    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-784v2" podStartSLOduration=2.50133811 podStartE2EDuration="6.505423226s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.004085116s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:34:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:34:43.504700753 +0000 UTC m=+2057.992068154" watchObservedRunningTime="2026-07-30 17:34:43.505423226 +0000 UTC m=+2057.992790632"
Jul 30 17:34:44.461350 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:44.461310    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-784v2" podUID="084bf132-1604-41f5-9b85-dcff676b84e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 30 17:34:44.461738 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:44.461420    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-hntx2" podUID="af5aecee-8785-40df-a16d-5361f4aa8a9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 30 17:34:49.466052 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:49.465949    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-hntx2"
Jul 30 17:34:49.466518 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:49.466418    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-784v2"
Jul 30 17:34:49.466518 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:49.466476    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-hntx2" podUID="af5aecee-8785-40df-a16d-5361f4aa8a9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 30 17:34:49.466899 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:49.466871    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-784v2" podUID="084bf132-1604-41f5-9b85-dcff676b84e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 30 17:34:59.467389 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:59.467349    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-hntx2" podUID="af5aecee-8785-40df-a16d-5361f4aa8a9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 30 17:34:59.467776 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:34:59.467349    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-784v2" podUID="084bf132-1604-41f5-9b85-dcff676b84e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 30 17:35:09.466723 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:35:09.466682    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-hntx2" podUID="af5aecee-8785-40df-a16d-5361f4aa8a9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 30 17:35:09.467222 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:35:09.466825    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-784v2" podUID="084bf132-1604-41f5-9b85-dcff676b84e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 30 17:35:19.466759 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:35:19.466714    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-hntx2" podUID="af5aecee-8785-40df-a16d-5361f4aa8a9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 30 17:35:19.467168 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:35:19.466840    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-784v2" podUID="084bf132-1604-41f5-9b85-dcff676b84e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 30 17:35:29.466839 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:35:29.466802    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-784v2" podUID="084bf132-1604-41f5-9b85-dcff676b84e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 30 17:35:29.467247 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:35:29.466802    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-hntx2" podUID="af5aecee-8785-40df-a16d-5361f4aa8a9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 30 17:35:39.467226 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:35:39.467186    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-hntx2" podUID="af5aecee-8785-40df-a16d-5361f4aa8a9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 30 17:35:39.467635 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:35:39.467187    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-784v2" podUID="084bf132-1604-41f5-9b85-dcff676b84e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 30 17:35:49.467312 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:35:49.467274    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-hntx2" podUID="af5aecee-8785-40df-a16d-5361f4aa8a9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 30 17:35:49.467773 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:35:49.467275    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-784v2" podUID="084bf132-1604-41f5-9b85-dcff676b84e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 30 17:35:54.055395 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:35:54.055345    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-hntx2" podUID="af5aecee-8785-40df-a16d-5361f4aa8a9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 30 17:35:59.467937 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:35:59.467905    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-784v2"
Jul 30 17:36:04.060913 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:04.060884    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-hntx2"
Jul 30 17:36:07.320167 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:07.320129    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-hntx2"]
Jul 30 17:36:07.320603 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:07.320418    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-hntx2" podUID="af5aecee-8785-40df-a16d-5361f4aa8a9e" containerName="kserve-container" containerID="cri-o://12818f4774854176b433704d6be440bfe3e1166deb34f50ff0a89b5610dd1013" gracePeriod=30
Jul 30 17:36:07.320603 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:07.320501    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-hntx2" podUID="af5aecee-8785-40df-a16d-5361f4aa8a9e" containerName="kube-rbac-proxy" containerID="cri-o://dd83b4829088514f025df178d2a99ac1ee828be8b31ef3cc5d6655b9505e15da" gracePeriod=30
Jul 30 17:36:07.335437 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:07.335412    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:36:07.351572 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:07.351543    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-784v2"]
Jul 30 17:36:07.351769 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:07.351748    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-784v2" podUID="084bf132-1604-41f5-9b85-dcff676b84e9" containerName="kserve-container" containerID="cri-o://cdd86790b98f3491684e57a8769c578b2b128005aebdba866247b363f062e6c8" gracePeriod=30
Jul 30 17:36:07.351865 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:07.351792    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-784v2" podUID="084bf132-1604-41f5-9b85-dcff676b84e9" containerName="kube-rbac-proxy" containerID="cri-o://0ff6c0bc959d629eb6ac36949bfb38ce42f16f5ea7a4361774a9bb334bab2bd7" gracePeriod=30
Jul 30 17:36:07.431122 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:07.431090    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:36:07.855161 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:07.855130    2555 generic.go:356] "Generic (PLEG): container finished" podID="af5aecee-8785-40df-a16d-5361f4aa8a9e" containerID="dd83b4829088514f025df178d2a99ac1ee828be8b31ef3cc5d6655b9505e15da" exitCode=2
Jul 30 17:36:07.855354 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:07.855202    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-hntx2" event={"ID":"af5aecee-8785-40df-a16d-5361f4aa8a9e","Type":"ContainerDied","Data":"dd83b4829088514f025df178d2a99ac1ee828be8b31ef3cc5d6655b9505e15da"}
Jul 30 17:36:07.857331 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:07.857309    2555 generic.go:356] "Generic (PLEG): container finished" podID="084bf132-1604-41f5-9b85-dcff676b84e9" containerID="0ff6c0bc959d629eb6ac36949bfb38ce42f16f5ea7a4361774a9bb334bab2bd7" exitCode=2
Jul 30 17:36:07.857444 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:07.857381    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-784v2" event={"ID":"084bf132-1604-41f5-9b85-dcff676b84e9","Type":"ContainerDied","Data":"0ff6c0bc959d629eb6ac36949bfb38ce42f16f5ea7a4361774a9bb334bab2bd7"}
Jul 30 17:36:09.461884 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:09.461844    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-784v2" podUID="084bf132-1604-41f5-9b85-dcff676b84e9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.40:8643/healthz\": dial tcp 10.133.0.40:8643: connect: connection refused"
Jul 30 17:36:09.462313 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:09.461844    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-hntx2" podUID="af5aecee-8785-40df-a16d-5361f4aa8a9e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.39:8643/healthz\": dial tcp 10.133.0.39:8643: connect: connection refused"
Jul 30 17:36:09.467207 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:09.467186    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-784v2" podUID="084bf132-1604-41f5-9b85-dcff676b84e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 30 17:36:11.878507 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:11.878473    2555 generic.go:356] "Generic (PLEG): container finished" podID="af5aecee-8785-40df-a16d-5361f4aa8a9e" containerID="12818f4774854176b433704d6be440bfe3e1166deb34f50ff0a89b5610dd1013" exitCode=0
Jul 30 17:36:11.878937 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:11.878546    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-hntx2" event={"ID":"af5aecee-8785-40df-a16d-5361f4aa8a9e","Type":"ContainerDied","Data":"12818f4774854176b433704d6be440bfe3e1166deb34f50ff0a89b5610dd1013"}
Jul 30 17:36:11.994067 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:11.994036    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-hntx2"
Jul 30 17:36:11.998389 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:11.998359    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af5aecee-8785-40df-a16d-5361f4aa8a9e-proxy-tls\") pod \"af5aecee-8785-40df-a16d-5361f4aa8a9e\" (UID: \"af5aecee-8785-40df-a16d-5361f4aa8a9e\") "
Jul 30 17:36:11.998541 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:11.998422    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/af5aecee-8785-40df-a16d-5361f4aa8a9e-kserve-provision-location\") pod \"af5aecee-8785-40df-a16d-5361f4aa8a9e\" (UID: \"af5aecee-8785-40df-a16d-5361f4aa8a9e\") "
Jul 30 17:36:11.998541 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:11.998470    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/af5aecee-8785-40df-a16d-5361f4aa8a9e-cabundle-cert\") pod \"af5aecee-8785-40df-a16d-5361f4aa8a9e\" (UID: \"af5aecee-8785-40df-a16d-5361f4aa8a9e\") "
Jul 30 17:36:11.998541 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:11.998519    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/af5aecee-8785-40df-a16d-5361f4aa8a9e-kube-rbac-proxy-sar-config\") pod \"af5aecee-8785-40df-a16d-5361f4aa8a9e\" (UID: \"af5aecee-8785-40df-a16d-5361f4aa8a9e\") "
Jul 30 17:36:11.998712 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:11.998575    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d88qs\" (UniqueName: \"kubernetes.io/projected/af5aecee-8785-40df-a16d-5361f4aa8a9e-kube-api-access-d88qs\") pod \"af5aecee-8785-40df-a16d-5361f4aa8a9e\" (UID: \"af5aecee-8785-40df-a16d-5361f4aa8a9e\") "
Jul 30 17:36:11.998789 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:11.998763    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af5aecee-8785-40df-a16d-5361f4aa8a9e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "af5aecee-8785-40df-a16d-5361f4aa8a9e" (UID: "af5aecee-8785-40df-a16d-5361f4aa8a9e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 17:36:11.998881 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:11.998866    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/af5aecee-8785-40df-a16d-5361f4aa8a9e-kserve-provision-location\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:36:11.998917 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:11.998895    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af5aecee-8785-40df-a16d-5361f4aa8a9e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "af5aecee-8785-40df-a16d-5361f4aa8a9e" (UID: "af5aecee-8785-40df-a16d-5361f4aa8a9e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:36:11.999310 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:11.999284    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af5aecee-8785-40df-a16d-5361f4aa8a9e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "af5aecee-8785-40df-a16d-5361f4aa8a9e" (UID: "af5aecee-8785-40df-a16d-5361f4aa8a9e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:36:12.000699 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:12.000679    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af5aecee-8785-40df-a16d-5361f4aa8a9e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "af5aecee-8785-40df-a16d-5361f4aa8a9e" (UID: "af5aecee-8785-40df-a16d-5361f4aa8a9e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:36:12.000868 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:12.000845    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af5aecee-8785-40df-a16d-5361f4aa8a9e-kube-api-access-d88qs" (OuterVolumeSpecName: "kube-api-access-d88qs") pod "af5aecee-8785-40df-a16d-5361f4aa8a9e" (UID: "af5aecee-8785-40df-a16d-5361f4aa8a9e"). InnerVolumeSpecName "kube-api-access-d88qs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 17:36:12.099381 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:12.099359    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/af5aecee-8785-40df-a16d-5361f4aa8a9e-cabundle-cert\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:36:12.099381 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:12.099381    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/af5aecee-8785-40df-a16d-5361f4aa8a9e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:36:12.099381 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:12.099392    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d88qs\" (UniqueName: \"kubernetes.io/projected/af5aecee-8785-40df-a16d-5361f4aa8a9e-kube-api-access-d88qs\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:36:12.099578 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:12.099401    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af5aecee-8785-40df-a16d-5361f4aa8a9e-proxy-tls\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:36:12.105609 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:12.105590    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-784v2"
Jul 30 17:36:12.200477 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:12.200444    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9znss\" (UniqueName: \"kubernetes.io/projected/084bf132-1604-41f5-9b85-dcff676b84e9-kube-api-access-9znss\") pod \"084bf132-1604-41f5-9b85-dcff676b84e9\" (UID: \"084bf132-1604-41f5-9b85-dcff676b84e9\") "
Jul 30 17:36:12.200687 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:12.200515    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/084bf132-1604-41f5-9b85-dcff676b84e9-proxy-tls\") pod \"084bf132-1604-41f5-9b85-dcff676b84e9\" (UID: \"084bf132-1604-41f5-9b85-dcff676b84e9\") "
Jul 30 17:36:12.200687 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:12.200543    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/084bf132-1604-41f5-9b85-dcff676b84e9-cabundle-cert\") pod \"084bf132-1604-41f5-9b85-dcff676b84e9\" (UID: \"084bf132-1604-41f5-9b85-dcff676b84e9\") "
Jul 30 17:36:12.200687 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:12.200596    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/084bf132-1604-41f5-9b85-dcff676b84e9-kserve-provision-location\") pod \"084bf132-1604-41f5-9b85-dcff676b84e9\" (UID: \"084bf132-1604-41f5-9b85-dcff676b84e9\") "
Jul 30 17:36:12.200860 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:12.200716    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/084bf132-1604-41f5-9b85-dcff676b84e9-kube-rbac-proxy-sar-config\") pod \"084bf132-1604-41f5-9b85-dcff676b84e9\" (UID: \"084bf132-1604-41f5-9b85-dcff676b84e9\") "
Jul 30 17:36:12.200962 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:12.200937    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/084bf132-1604-41f5-9b85-dcff676b84e9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "084bf132-1604-41f5-9b85-dcff676b84e9" (UID: "084bf132-1604-41f5-9b85-dcff676b84e9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 17:36:12.201071 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:12.201035    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/084bf132-1604-41f5-9b85-dcff676b84e9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "084bf132-1604-41f5-9b85-dcff676b84e9" (UID: "084bf132-1604-41f5-9b85-dcff676b84e9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:36:12.201120 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:12.201074    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/084bf132-1604-41f5-9b85-dcff676b84e9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "084bf132-1604-41f5-9b85-dcff676b84e9" (UID: "084bf132-1604-41f5-9b85-dcff676b84e9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:36:12.202593 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:12.202568    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/084bf132-1604-41f5-9b85-dcff676b84e9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "084bf132-1604-41f5-9b85-dcff676b84e9" (UID: "084bf132-1604-41f5-9b85-dcff676b84e9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:36:12.202697 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:12.202619    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/084bf132-1604-41f5-9b85-dcff676b84e9-kube-api-access-9znss" (OuterVolumeSpecName: "kube-api-access-9znss") pod "084bf132-1604-41f5-9b85-dcff676b84e9" (UID: "084bf132-1604-41f5-9b85-dcff676b84e9"). InnerVolumeSpecName "kube-api-access-9znss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 17:36:12.301721 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:12.301689    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/084bf132-1604-41f5-9b85-dcff676b84e9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:36:12.301721 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:12.301720    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9znss\" (UniqueName: \"kubernetes.io/projected/084bf132-1604-41f5-9b85-dcff676b84e9-kube-api-access-9znss\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:36:12.301721 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:12.301730    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/084bf132-1604-41f5-9b85-dcff676b84e9-proxy-tls\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:36:12.301945 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:12.301740    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/084bf132-1604-41f5-9b85-dcff676b84e9-cabundle-cert\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:36:12.301945 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:12.301749    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/084bf132-1604-41f5-9b85-dcff676b84e9-kserve-provision-location\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:36:12.885694 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:12.885664    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-hntx2"
Jul 30 17:36:12.885694 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:12.885686    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-hntx2" event={"ID":"af5aecee-8785-40df-a16d-5361f4aa8a9e","Type":"ContainerDied","Data":"34cb87bfb36e8720a13695f0052f0a3740c48f32e43c737ef6f088e2e87155b3"}
Jul 30 17:36:12.886219 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:12.885751    2555 scope.go:117] "RemoveContainer" containerID="dd83b4829088514f025df178d2a99ac1ee828be8b31ef3cc5d6655b9505e15da"
Jul 30 17:36:12.887608 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:12.887585    2555 generic.go:356] "Generic (PLEG): container finished" podID="084bf132-1604-41f5-9b85-dcff676b84e9" containerID="cdd86790b98f3491684e57a8769c578b2b128005aebdba866247b363f062e6c8" exitCode=0
Jul 30 17:36:12.887738 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:12.887618    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-784v2" event={"ID":"084bf132-1604-41f5-9b85-dcff676b84e9","Type":"ContainerDied","Data":"cdd86790b98f3491684e57a8769c578b2b128005aebdba866247b363f062e6c8"}
Jul 30 17:36:12.887738 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:12.887655    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-784v2"
Jul 30 17:36:12.887738 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:12.887684    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-784v2" event={"ID":"084bf132-1604-41f5-9b85-dcff676b84e9","Type":"ContainerDied","Data":"38616051f852ab62d7a811505656acc8925a4848a97c4238b7286ac03b44c7bf"}
Jul 30 17:36:12.894265 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:12.894248    2555 scope.go:117] "RemoveContainer" containerID="12818f4774854176b433704d6be440bfe3e1166deb34f50ff0a89b5610dd1013"
Jul 30 17:36:12.901716 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:12.901700    2555 scope.go:117] "RemoveContainer" containerID="8e8aea58c0e6fc9b42881fc56aed3809047ba859ffef420473856cc00575d3d9"
Jul 30 17:36:12.908500 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:12.908477    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-hntx2"]
Jul 30 17:36:12.910145 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:12.910124    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-77665d5899-hntx2"]
Jul 30 17:36:12.910633 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:12.910619    2555 scope.go:117] "RemoveContainer" containerID="0ff6c0bc959d629eb6ac36949bfb38ce42f16f5ea7a4361774a9bb334bab2bd7"
Jul 30 17:36:12.918245 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:12.918226    2555 scope.go:117] "RemoveContainer" containerID="cdd86790b98f3491684e57a8769c578b2b128005aebdba866247b363f062e6c8"
Jul 30 17:36:12.925114 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:12.925093    2555 scope.go:117] "RemoveContainer" containerID="ff4c8f6c7d02d1deec8920a80d27b5f0c11debcfdfcfefa45a205b8439f2e73a"
Jul 30 17:36:12.928982 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:12.928965    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-784v2"]
Jul 30 17:36:12.933186 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:12.933170    2555 scope.go:117] "RemoveContainer" containerID="0ff6c0bc959d629eb6ac36949bfb38ce42f16f5ea7a4361774a9bb334bab2bd7"
Jul 30 17:36:12.933494 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:36:12.933472    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ff6c0bc959d629eb6ac36949bfb38ce42f16f5ea7a4361774a9bb334bab2bd7\": container with ID starting with 0ff6c0bc959d629eb6ac36949bfb38ce42f16f5ea7a4361774a9bb334bab2bd7 not found: ID does not exist" containerID="0ff6c0bc959d629eb6ac36949bfb38ce42f16f5ea7a4361774a9bb334bab2bd7"
Jul 30 17:36:12.933550 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:12.933496    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-9b8c9778-784v2"]
Jul 30 17:36:12.933550 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:12.933503    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ff6c0bc959d629eb6ac36949bfb38ce42f16f5ea7a4361774a9bb334bab2bd7"} err="failed to get container status \"0ff6c0bc959d629eb6ac36949bfb38ce42f16f5ea7a4361774a9bb334bab2bd7\": rpc error: code = NotFound desc = could not find container \"0ff6c0bc959d629eb6ac36949bfb38ce42f16f5ea7a4361774a9bb334bab2bd7\": container with ID starting with 0ff6c0bc959d629eb6ac36949bfb38ce42f16f5ea7a4361774a9bb334bab2bd7 not found: ID does not exist"
Jul 30 17:36:12.933550 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:12.933521    2555 scope.go:117] "RemoveContainer" containerID="cdd86790b98f3491684e57a8769c578b2b128005aebdba866247b363f062e6c8"
Jul 30 17:36:12.933746 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:36:12.933729    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdd86790b98f3491684e57a8769c578b2b128005aebdba866247b363f062e6c8\": container with ID starting with cdd86790b98f3491684e57a8769c578b2b128005aebdba866247b363f062e6c8 not found: ID does not exist" containerID="cdd86790b98f3491684e57a8769c578b2b128005aebdba866247b363f062e6c8"
Jul 30 17:36:12.933790 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:12.933753    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdd86790b98f3491684e57a8769c578b2b128005aebdba866247b363f062e6c8"} err="failed to get container status \"cdd86790b98f3491684e57a8769c578b2b128005aebdba866247b363f062e6c8\": rpc error: code = NotFound desc = could not find container \"cdd86790b98f3491684e57a8769c578b2b128005aebdba866247b363f062e6c8\": container with ID starting with cdd86790b98f3491684e57a8769c578b2b128005aebdba866247b363f062e6c8 not found: ID does not exist"
Jul 30 17:36:12.933790 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:12.933769    2555 scope.go:117] "RemoveContainer" containerID="ff4c8f6c7d02d1deec8920a80d27b5f0c11debcfdfcfefa45a205b8439f2e73a"
Jul 30 17:36:12.934031 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:36:12.933993    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff4c8f6c7d02d1deec8920a80d27b5f0c11debcfdfcfefa45a205b8439f2e73a\": container with ID starting with ff4c8f6c7d02d1deec8920a80d27b5f0c11debcfdfcfefa45a205b8439f2e73a not found: ID does not exist" containerID="ff4c8f6c7d02d1deec8920a80d27b5f0c11debcfdfcfefa45a205b8439f2e73a"
Jul 30 17:36:12.934107 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:12.934033    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff4c8f6c7d02d1deec8920a80d27b5f0c11debcfdfcfefa45a205b8439f2e73a"} err="failed to get container status \"ff4c8f6c7d02d1deec8920a80d27b5f0c11debcfdfcfefa45a205b8439f2e73a\": rpc error: code = NotFound desc = could not find container \"ff4c8f6c7d02d1deec8920a80d27b5f0c11debcfdfcfefa45a205b8439f2e73a\": container with ID starting with ff4c8f6c7d02d1deec8920a80d27b5f0c11debcfdfcfefa45a205b8439f2e73a not found: ID does not exist"
Jul 30 17:36:13.670680 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:13.670651    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:36:13.773928 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:13.773897    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:36:14.061125 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:14.061091    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="084bf132-1604-41f5-9b85-dcff676b84e9" path="/var/lib/kubelet/pods/084bf132-1604-41f5-9b85-dcff676b84e9/volumes"
Jul 30 17:36:14.061597 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:14.061584    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af5aecee-8785-40df-a16d-5361f4aa8a9e" path="/var/lib/kubelet/pods/af5aecee-8785-40df-a16d-5361f4aa8a9e/volumes"
Jul 30 17:36:17.553317 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:17.553234    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:36:17.608834 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:17.608796    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-gb6cb"]
Jul 30 17:36:17.609179 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:17.609165    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="084bf132-1604-41f5-9b85-dcff676b84e9" containerName="kube-rbac-proxy"
Jul 30 17:36:17.609282 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:17.609181    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="084bf132-1604-41f5-9b85-dcff676b84e9" containerName="kube-rbac-proxy"
Jul 30 17:36:17.609282 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:17.609196    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af5aecee-8785-40df-a16d-5361f4aa8a9e" containerName="kube-rbac-proxy"
Jul 30 17:36:17.609282 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:17.609203    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="af5aecee-8785-40df-a16d-5361f4aa8a9e" containerName="kube-rbac-proxy"
Jul 30 17:36:17.609282 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:17.609218    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af5aecee-8785-40df-a16d-5361f4aa8a9e" containerName="storage-initializer"
Jul 30 17:36:17.609282 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:17.609223    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="af5aecee-8785-40df-a16d-5361f4aa8a9e" containerName="storage-initializer"
Jul 30 17:36:17.609282 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:17.609231    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="084bf132-1604-41f5-9b85-dcff676b84e9" containerName="kserve-container"
Jul 30 17:36:17.609282 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:17.609237    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="084bf132-1604-41f5-9b85-dcff676b84e9" containerName="kserve-container"
Jul 30 17:36:17.609282 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:17.609245    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="084bf132-1604-41f5-9b85-dcff676b84e9" containerName="storage-initializer"
Jul 30 17:36:17.609282 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:17.609250    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="084bf132-1604-41f5-9b85-dcff676b84e9" containerName="storage-initializer"
Jul 30 17:36:17.609282 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:17.609257    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af5aecee-8785-40df-a16d-5361f4aa8a9e" containerName="kserve-container"
Jul 30 17:36:17.609282 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:17.609262    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="af5aecee-8785-40df-a16d-5361f4aa8a9e" containerName="kserve-container"
Jul 30 17:36:17.609652 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:17.609317    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="084bf132-1604-41f5-9b85-dcff676b84e9" containerName="kube-rbac-proxy"
Jul 30 17:36:17.609652 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:17.609327    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="af5aecee-8785-40df-a16d-5361f4aa8a9e" containerName="kserve-container"
Jul 30 17:36:17.609652 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:17.609335    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="084bf132-1604-41f5-9b85-dcff676b84e9" containerName="kserve-container"
Jul 30 17:36:17.609652 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:17.609341    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="af5aecee-8785-40df-a16d-5361f4aa8a9e" containerName="kube-rbac-proxy"
Jul 30 17:36:17.614334 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:17.614313    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-gb6cb"
Jul 30 17:36:17.617249 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:17.617222    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 30 17:36:17.617249 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:17.617223    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 30 17:36:17.617509 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:17.617223    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 30 17:36:17.618296 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:17.618275    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-plw5t\""
Jul 30 17:36:17.618415 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:17.618304    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-predictor-serving-cert\""
Jul 30 17:36:17.618415 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:17.618329    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-kube-rbac-proxy-sar-config\""
Jul 30 17:36:17.618415 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:17.618370    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 30 17:36:17.623414 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:17.623393    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-gb6cb"]
Jul 30 17:36:17.629154 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:17.629130    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:36:17.639381 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:17.639353    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd4j5\" (UniqueName: \"kubernetes.io/projected/3f08cda5-90a1-431d-b6ae-85f6cf073099-kube-api-access-xd4j5\") pod \"isvc-lightgbm-predictor-54f85ddf88-gb6cb\" (UID: \"3f08cda5-90a1-431d-b6ae-85f6cf073099\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-gb6cb"
Jul 30 17:36:17.639498 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:17.639396    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3f08cda5-90a1-431d-b6ae-85f6cf073099-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-gb6cb\" (UID: \"3f08cda5-90a1-431d-b6ae-85f6cf073099\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-gb6cb"
Jul 30 17:36:17.639498 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:17.639481    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3f08cda5-90a1-431d-b6ae-85f6cf073099-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-gb6cb\" (UID: \"3f08cda5-90a1-431d-b6ae-85f6cf073099\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-gb6cb"
Jul 30 17:36:17.639631 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:17.639558    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3f08cda5-90a1-431d-b6ae-85f6cf073099-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-gb6cb\" (UID: \"3f08cda5-90a1-431d-b6ae-85f6cf073099\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-gb6cb"
Jul 30 17:36:17.639631 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:17.639598    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f08cda5-90a1-431d-b6ae-85f6cf073099-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-gb6cb\" (UID: \"3f08cda5-90a1-431d-b6ae-85f6cf073099\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-gb6cb"
Jul 30 17:36:17.740636 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:17.740590    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f08cda5-90a1-431d-b6ae-85f6cf073099-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-gb6cb\" (UID: \"3f08cda5-90a1-431d-b6ae-85f6cf073099\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-gb6cb"
Jul 30 17:36:17.740815 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:17.740657    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd4j5\" (UniqueName: \"kubernetes.io/projected/3f08cda5-90a1-431d-b6ae-85f6cf073099-kube-api-access-xd4j5\") pod \"isvc-lightgbm-predictor-54f85ddf88-gb6cb\" (UID: \"3f08cda5-90a1-431d-b6ae-85f6cf073099\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-gb6cb"
Jul 30 17:36:17.740815 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:17.740676    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3f08cda5-90a1-431d-b6ae-85f6cf073099-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-gb6cb\" (UID: \"3f08cda5-90a1-431d-b6ae-85f6cf073099\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-gb6cb"
Jul 30 17:36:17.740815 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:17.740713    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3f08cda5-90a1-431d-b6ae-85f6cf073099-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-gb6cb\" (UID: \"3f08cda5-90a1-431d-b6ae-85f6cf073099\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-gb6cb"
Jul 30 17:36:17.740815 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:17.740760    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3f08cda5-90a1-431d-b6ae-85f6cf073099-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-gb6cb\" (UID: \"3f08cda5-90a1-431d-b6ae-85f6cf073099\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-gb6cb"
Jul 30 17:36:17.741102 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:17.741084    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f08cda5-90a1-431d-b6ae-85f6cf073099-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-gb6cb\" (UID: \"3f08cda5-90a1-431d-b6ae-85f6cf073099\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-gb6cb"
Jul 30 17:36:17.741463 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:17.741445    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3f08cda5-90a1-431d-b6ae-85f6cf073099-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-gb6cb\" (UID: \"3f08cda5-90a1-431d-b6ae-85f6cf073099\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-gb6cb"
Jul 30 17:36:17.741463 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:17.741443    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3f08cda5-90a1-431d-b6ae-85f6cf073099-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-gb6cb\" (UID: \"3f08cda5-90a1-431d-b6ae-85f6cf073099\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-gb6cb"
Jul 30 17:36:17.743177 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:17.743161    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3f08cda5-90a1-431d-b6ae-85f6cf073099-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-gb6cb\" (UID: \"3f08cda5-90a1-431d-b6ae-85f6cf073099\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-gb6cb"
Jul 30 17:36:17.751113 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:17.751083    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd4j5\" (UniqueName: \"kubernetes.io/projected/3f08cda5-90a1-431d-b6ae-85f6cf073099-kube-api-access-xd4j5\") pod \"isvc-lightgbm-predictor-54f85ddf88-gb6cb\" (UID: \"3f08cda5-90a1-431d-b6ae-85f6cf073099\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-gb6cb"
Jul 30 17:36:17.926471 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:17.926384    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-gb6cb"
Jul 30 17:36:18.060822 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:36:18.060794    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f08cda5_90a1_431d_b6ae_85f6cf073099.slice/crio-5b7c94e47c929320ea722c71d6dbf55184117f84874dbdac210174006e85d85e WatchSource:0}: Error finding container 5b7c94e47c929320ea722c71d6dbf55184117f84874dbdac210174006e85d85e: Status 404 returned error can't find the container with id 5b7c94e47c929320ea722c71d6dbf55184117f84874dbdac210174006e85d85e
Jul 30 17:36:18.061335 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:18.061313    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-gb6cb"]
Jul 30 17:36:18.921949 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:18.921913    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-gb6cb" event={"ID":"3f08cda5-90a1-431d-b6ae-85f6cf073099","Type":"ContainerStarted","Data":"877a526b78b3de2f09eb6cd9bc7b8947233118b001580e013b4ee8105b26fd03"}
Jul 30 17:36:18.922358 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:18.921996    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-gb6cb" event={"ID":"3f08cda5-90a1-431d-b6ae-85f6cf073099","Type":"ContainerStarted","Data":"5b7c94e47c929320ea722c71d6dbf55184117f84874dbdac210174006e85d85e"}
Jul 30 17:36:22.942801 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:22.942770    2555 generic.go:356] "Generic (PLEG): container finished" podID="3f08cda5-90a1-431d-b6ae-85f6cf073099" containerID="877a526b78b3de2f09eb6cd9bc7b8947233118b001580e013b4ee8105b26fd03" exitCode=0
Jul 30 17:36:22.943275 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:22.942846    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-gb6cb" event={"ID":"3f08cda5-90a1-431d-b6ae-85f6cf073099","Type":"ContainerDied","Data":"877a526b78b3de2f09eb6cd9bc7b8947233118b001580e013b4ee8105b26fd03"}
Jul 30 17:36:22.944145 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:22.944128    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 30 17:36:29.983873 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:29.983837    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-gb6cb" event={"ID":"3f08cda5-90a1-431d-b6ae-85f6cf073099","Type":"ContainerStarted","Data":"8e5ff985cb4e282cda05dafa97c0e4d28f6574fba86c198ee71af6472a7e7226"}
Jul 30 17:36:29.984288 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:29.983901    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-gb6cb" event={"ID":"3f08cda5-90a1-431d-b6ae-85f6cf073099","Type":"ContainerStarted","Data":"ff09e6801973f16d83711ee0fbf5e5cc858b547399a48ed837d5d1b3da7626c0"}
Jul 30 17:36:29.984288 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:29.984106    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-gb6cb"
Jul 30 17:36:30.011535 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:30.011484    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-gb6cb" podStartSLOduration=2.504831024 podStartE2EDuration="13.011468646s" totalImagesPullingTime="6.67005972s" totalInitContainerRuntime="3.836577902s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:36:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:36:30.009770282 +0000 UTC m=+2164.497137688" watchObservedRunningTime="2026-07-30 17:36:30.011468646 +0000 UTC m=+2164.498836046"
Jul 30 17:36:30.988692 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:30.988658    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-gb6cb"
Jul 30 17:36:30.990091 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:30.990065    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-gb6cb" podUID="3f08cda5-90a1-431d-b6ae-85f6cf073099" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 30 17:36:31.993066 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:31.993029    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-gb6cb" podUID="3f08cda5-90a1-431d-b6ae-85f6cf073099" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 30 17:36:36.999641 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:36.999610    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-gb6cb"
Jul 30 17:36:37.000283 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:37.000255    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-gb6cb" podUID="3f08cda5-90a1-431d-b6ae-85f6cf073099" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 30 17:36:47.000893 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:47.000847    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-gb6cb" podUID="3f08cda5-90a1-431d-b6ae-85f6cf073099" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 30 17:36:57.000628 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:36:57.000592    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-gb6cb" podUID="3f08cda5-90a1-431d-b6ae-85f6cf073099" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 30 17:37:07.000329 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:37:07.000283    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-gb6cb" podUID="3f08cda5-90a1-431d-b6ae-85f6cf073099" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 30 17:37:17.001029 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:37:17.000964    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-gb6cb" podUID="3f08cda5-90a1-431d-b6ae-85f6cf073099" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 30 17:37:27.001128 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:37:27.001087    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-gb6cb" podUID="3f08cda5-90a1-431d-b6ae-85f6cf073099" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 30 17:37:37.000502 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:37:37.000440    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-gb6cb" podUID="3f08cda5-90a1-431d-b6ae-85f6cf073099" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 30 17:37:47.001171 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:37:47.001139    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-gb6cb"
Jul 30 17:37:47.900911 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:37:47.900837    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-gb6cb"]
Jul 30 17:37:47.901141 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:37:47.901110    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-gb6cb" podUID="3f08cda5-90a1-431d-b6ae-85f6cf073099" containerName="kserve-container" containerID="cri-o://ff09e6801973f16d83711ee0fbf5e5cc858b547399a48ed837d5d1b3da7626c0" gracePeriod=30
Jul 30 17:37:47.901292 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:37:47.901162    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-gb6cb" podUID="3f08cda5-90a1-431d-b6ae-85f6cf073099" containerName="kube-rbac-proxy" containerID="cri-o://8e5ff985cb4e282cda05dafa97c0e4d28f6574fba86c198ee71af6472a7e7226" gracePeriod=30
Jul 30 17:37:47.959887 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:37:47.959852    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:37:48.346925 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:37:48.346896    2555 generic.go:356] "Generic (PLEG): container finished" podID="3f08cda5-90a1-431d-b6ae-85f6cf073099" containerID="8e5ff985cb4e282cda05dafa97c0e4d28f6574fba86c198ee71af6472a7e7226" exitCode=2
Jul 30 17:37:48.347303 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:37:48.346966    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-gb6cb" event={"ID":"3f08cda5-90a1-431d-b6ae-85f6cf073099","Type":"ContainerDied","Data":"8e5ff985cb4e282cda05dafa97c0e4d28f6574fba86c198ee71af6472a7e7226"}
Jul 30 17:37:51.994205 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:37:51.994161    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-gb6cb" podUID="3f08cda5-90a1-431d-b6ae-85f6cf073099" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.41:8643/healthz\": dial tcp 10.133.0.41:8643: connect: connection refused"
Jul 30 17:37:52.994482 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:37:52.994445    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:37:53.060032 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:37:53.059971    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-m5rfb"]
Jul 30 17:37:53.063505 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:37:53.063490    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-m5rfb"
Jul 30 17:37:53.066166 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:37:53.066143    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-predictor-serving-cert\""
Jul 30 17:37:53.066287 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:37:53.066168    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\""
Jul 30 17:37:53.075326 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:37:53.075304    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-m5rfb"]
Jul 30 17:37:53.171660 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:37:53.171623    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/418014eb-a790-40be-93c6-6ea289e644e1-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-m5rfb\" (UID: \"418014eb-a790-40be-93c6-6ea289e644e1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-m5rfb"
Jul 30 17:37:53.171660 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:37:53.171667    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/418014eb-a790-40be-93c6-6ea289e644e1-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-m5rfb\" (UID: \"418014eb-a790-40be-93c6-6ea289e644e1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-m5rfb"
Jul 30 17:37:53.171870 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:37:53.171685    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/418014eb-a790-40be-93c6-6ea289e644e1-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-m5rfb\" (UID: \"418014eb-a790-40be-93c6-6ea289e644e1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-m5rfb"
Jul 30 17:37:53.171870 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:37:53.171757    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/418014eb-a790-40be-93c6-6ea289e644e1-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-m5rfb\" (UID: \"418014eb-a790-40be-93c6-6ea289e644e1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-m5rfb"
Jul 30 17:37:53.171870 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:37:53.171785    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6q4z\" (UniqueName: \"kubernetes.io/projected/418014eb-a790-40be-93c6-6ea289e644e1-kube-api-access-b6q4z\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-m5rfb\" (UID: \"418014eb-a790-40be-93c6-6ea289e644e1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-m5rfb"
Jul 30 17:37:53.272739 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:37:53.272660    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/418014eb-a790-40be-93c6-6ea289e644e1-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-m5rfb\" (UID: \"418014eb-a790-40be-93c6-6ea289e644e1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-m5rfb"
Jul 30 17:37:53.272739 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:37:53.272707    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6q4z\" (UniqueName: \"kubernetes.io/projected/418014eb-a790-40be-93c6-6ea289e644e1-kube-api-access-b6q4z\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-m5rfb\" (UID: \"418014eb-a790-40be-93c6-6ea289e644e1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-m5rfb"
Jul 30 17:37:53.272955 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:37:53.272769    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/418014eb-a790-40be-93c6-6ea289e644e1-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-m5rfb\" (UID: \"418014eb-a790-40be-93c6-6ea289e644e1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-m5rfb"
Jul 30 17:37:53.272955 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:37:53.272794    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/418014eb-a790-40be-93c6-6ea289e644e1-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-m5rfb\" (UID: \"418014eb-a790-40be-93c6-6ea289e644e1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-m5rfb"
Jul 30 17:37:53.272955 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:37:53.272813    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/418014eb-a790-40be-93c6-6ea289e644e1-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-m5rfb\" (UID: \"418014eb-a790-40be-93c6-6ea289e644e1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-m5rfb"
Jul 30 17:37:53.273283 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:37:53.273260    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/418014eb-a790-40be-93c6-6ea289e644e1-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-m5rfb\" (UID: \"418014eb-a790-40be-93c6-6ea289e644e1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-m5rfb"
Jul 30 17:37:53.273553 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:37:53.273533    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/418014eb-a790-40be-93c6-6ea289e644e1-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-m5rfb\" (UID: \"418014eb-a790-40be-93c6-6ea289e644e1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-m5rfb"
Jul 30 17:37:53.273553 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:37:53.273543    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/418014eb-a790-40be-93c6-6ea289e644e1-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-m5rfb\" (UID: \"418014eb-a790-40be-93c6-6ea289e644e1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-m5rfb"
Jul 30 17:37:53.275173 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:37:53.275140    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/418014eb-a790-40be-93c6-6ea289e644e1-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-m5rfb\" (UID: \"418014eb-a790-40be-93c6-6ea289e644e1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-m5rfb"
Jul 30 17:37:53.283086 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:37:53.283066    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6q4z\" (UniqueName: \"kubernetes.io/projected/418014eb-a790-40be-93c6-6ea289e644e1-kube-api-access-b6q4z\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-m5rfb\" (UID: \"418014eb-a790-40be-93c6-6ea289e644e1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-m5rfb"
Jul 30 17:37:53.307921 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:37:53.307896    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:37:53.374234 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:37:53.374197    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-m5rfb"
Jul 30 17:37:53.507857 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:37:53.507831    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-m5rfb"]
Jul 30 17:37:53.509963 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:37:53.509934    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod418014eb_a790_40be_93c6_6ea289e644e1.slice/crio-bb8f273c3eb1c0b4891ff8647beffae8e154153dc62d42678400226c012ec462 WatchSource:0}: Error finding container bb8f273c3eb1c0b4891ff8647beffae8e154153dc62d42678400226c012ec462: Status 404 returned error can't find the container with id bb8f273c3eb1c0b4891ff8647beffae8e154153dc62d42678400226c012ec462
Jul 30 17:37:54.375544 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:37:54.375517    2555 generic.go:356] "Generic (PLEG): container finished" podID="3f08cda5-90a1-431d-b6ae-85f6cf073099" containerID="ff09e6801973f16d83711ee0fbf5e5cc858b547399a48ed837d5d1b3da7626c0" exitCode=0
Jul 30 17:37:54.375955 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:37:54.375591    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-gb6cb" event={"ID":"3f08cda5-90a1-431d-b6ae-85f6cf073099","Type":"ContainerDied","Data":"ff09e6801973f16d83711ee0fbf5e5cc858b547399a48ed837d5d1b3da7626c0"}
Jul 30 17:37:54.380964 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:37:54.380932    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-m5rfb" event={"ID":"418014eb-a790-40be-93c6-6ea289e644e1","Type":"ContainerStarted","Data":"e6949377549660323cc1e670c4e5d3b96146d419ccd520f5a0ef7fa53a3d7c76"}
Jul 30 17:37:54.381119 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:37:54.381022    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-m5rfb" event={"ID":"418014eb-a790-40be-93c6-6ea289e644e1","Type":"ContainerStarted","Data":"bb8f273c3eb1c0b4891ff8647beffae8e154153dc62d42678400226c012ec462"}
Jul 30 17:37:54.458271 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:37:54.458242    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-gb6cb"
Jul 30 17:37:54.585825 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:37:54.585789    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f08cda5-90a1-431d-b6ae-85f6cf073099-kserve-provision-location\") pod \"3f08cda5-90a1-431d-b6ae-85f6cf073099\" (UID: \"3f08cda5-90a1-431d-b6ae-85f6cf073099\") "
Jul 30 17:37:54.585999 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:37:54.585835    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xd4j5\" (UniqueName: \"kubernetes.io/projected/3f08cda5-90a1-431d-b6ae-85f6cf073099-kube-api-access-xd4j5\") pod \"3f08cda5-90a1-431d-b6ae-85f6cf073099\" (UID: \"3f08cda5-90a1-431d-b6ae-85f6cf073099\") "
Jul 30 17:37:54.585999 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:37:54.585852    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3f08cda5-90a1-431d-b6ae-85f6cf073099-cabundle-cert\") pod \"3f08cda5-90a1-431d-b6ae-85f6cf073099\" (UID: \"3f08cda5-90a1-431d-b6ae-85f6cf073099\") "
Jul 30 17:37:54.585999 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:37:54.585933    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3f08cda5-90a1-431d-b6ae-85f6cf073099-proxy-tls\") pod \"3f08cda5-90a1-431d-b6ae-85f6cf073099\" (UID: \"3f08cda5-90a1-431d-b6ae-85f6cf073099\") "
Jul 30 17:37:54.585999 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:37:54.585951    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3f08cda5-90a1-431d-b6ae-85f6cf073099-kube-rbac-proxy-sar-config\") pod \"3f08cda5-90a1-431d-b6ae-85f6cf073099\" (UID: \"3f08cda5-90a1-431d-b6ae-85f6cf073099\") "
Jul 30 17:37:54.586292 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:37:54.586263    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f08cda5-90a1-431d-b6ae-85f6cf073099-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3f08cda5-90a1-431d-b6ae-85f6cf073099" (UID: "3f08cda5-90a1-431d-b6ae-85f6cf073099"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 17:37:54.586373 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:37:54.586328    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f08cda5-90a1-431d-b6ae-85f6cf073099-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3f08cda5-90a1-431d-b6ae-85f6cf073099" (UID: "3f08cda5-90a1-431d-b6ae-85f6cf073099"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:37:54.586373 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:37:54.586331    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f08cda5-90a1-431d-b6ae-85f6cf073099-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3f08cda5-90a1-431d-b6ae-85f6cf073099" (UID: "3f08cda5-90a1-431d-b6ae-85f6cf073099"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:37:54.588002 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:37:54.587982    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f08cda5-90a1-431d-b6ae-85f6cf073099-kube-api-access-xd4j5" (OuterVolumeSpecName: "kube-api-access-xd4j5") pod "3f08cda5-90a1-431d-b6ae-85f6cf073099" (UID: "3f08cda5-90a1-431d-b6ae-85f6cf073099"). InnerVolumeSpecName "kube-api-access-xd4j5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 17:37:54.588002 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:37:54.587991    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f08cda5-90a1-431d-b6ae-85f6cf073099-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3f08cda5-90a1-431d-b6ae-85f6cf073099" (UID: "3f08cda5-90a1-431d-b6ae-85f6cf073099"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:37:54.686571 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:37:54.686523    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3f08cda5-90a1-431d-b6ae-85f6cf073099-proxy-tls\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:37:54.686571 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:37:54.686565    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3f08cda5-90a1-431d-b6ae-85f6cf073099-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:37:54.686571 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:37:54.686576    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f08cda5-90a1-431d-b6ae-85f6cf073099-kserve-provision-location\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:37:54.686571 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:37:54.686587    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xd4j5\" (UniqueName: \"kubernetes.io/projected/3f08cda5-90a1-431d-b6ae-85f6cf073099-kube-api-access-xd4j5\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:37:54.686839 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:37:54.686596    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3f08cda5-90a1-431d-b6ae-85f6cf073099-cabundle-cert\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:37:55.388493 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:37:55.388470    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-gb6cb"
Jul 30 17:37:55.388493 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:37:55.388487    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-gb6cb" event={"ID":"3f08cda5-90a1-431d-b6ae-85f6cf073099","Type":"ContainerDied","Data":"5b7c94e47c929320ea722c71d6dbf55184117f84874dbdac210174006e85d85e"}
Jul 30 17:37:55.388974 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:37:55.388563    2555 scope.go:117] "RemoveContainer" containerID="8e5ff985cb4e282cda05dafa97c0e4d28f6574fba86c198ee71af6472a7e7226"
Jul 30 17:37:55.397753 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:37:55.397734    2555 scope.go:117] "RemoveContainer" containerID="ff09e6801973f16d83711ee0fbf5e5cc858b547399a48ed837d5d1b3da7626c0"
Jul 30 17:37:55.405040 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:37:55.405023    2555 scope.go:117] "RemoveContainer" containerID="877a526b78b3de2f09eb6cd9bc7b8947233118b001580e013b4ee8105b26fd03"
Jul 30 17:37:55.415593 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:37:55.415569    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-gb6cb"]
Jul 30 17:37:55.422144 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:37:55.422122    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-gb6cb"]
Jul 30 17:37:56.061685 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:37:56.061654    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f08cda5-90a1-431d-b6ae-85f6cf073099" path="/var/lib/kubelet/pods/3f08cda5-90a1-431d-b6ae-85f6cf073099/volumes"
Jul 30 17:37:57.401543 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:37:57.401509    2555 generic.go:356] "Generic (PLEG): container finished" podID="418014eb-a790-40be-93c6-6ea289e644e1" containerID="e6949377549660323cc1e670c4e5d3b96146d419ccd520f5a0ef7fa53a3d7c76" exitCode=0
Jul 30 17:37:57.401949 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:37:57.401582    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-m5rfb" event={"ID":"418014eb-a790-40be-93c6-6ea289e644e1","Type":"ContainerDied","Data":"e6949377549660323cc1e670c4e5d3b96146d419ccd520f5a0ef7fa53a3d7c76"}
Jul 30 17:37:58.408702 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:37:58.408672    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-m5rfb" event={"ID":"418014eb-a790-40be-93c6-6ea289e644e1","Type":"ContainerStarted","Data":"b19d314e1bbba5a305a2e35e5dbbe75d9c1a1015083bae1872739ab6c618c1cc"}
Jul 30 17:37:58.409122 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:37:58.408726    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-m5rfb" event={"ID":"418014eb-a790-40be-93c6-6ea289e644e1","Type":"ContainerStarted","Data":"01490904fef3e84e5b3840895251ba4d481990698261569d36eaccf22afd5781"}
Jul 30 17:37:58.409122 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:37:58.408995    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-m5rfb"
Jul 30 17:37:58.409236 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:37:58.409155    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-m5rfb"
Jul 30 17:37:58.410739 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:37:58.410710    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-m5rfb" podUID="418014eb-a790-40be-93c6-6ea289e644e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 30 17:37:58.429920 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:37:58.429869    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-m5rfb" podStartSLOduration=1.7038237349999998 podStartE2EDuration="5.429856096s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.726032361s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:37:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:37:58.427851037 +0000 UTC m=+2252.915218443" watchObservedRunningTime="2026-07-30 17:37:58.429856096 +0000 UTC m=+2252.917223542"
Jul 30 17:37:59.413686 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:37:59.413645    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-m5rfb" podUID="418014eb-a790-40be-93c6-6ea289e644e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 30 17:38:04.419680 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:38:04.419648    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-m5rfb"
Jul 30 17:38:04.420221 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:38:04.420195    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-m5rfb" podUID="418014eb-a790-40be-93c6-6ea289e644e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 30 17:38:14.420157 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:38:14.420116    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-m5rfb" podUID="418014eb-a790-40be-93c6-6ea289e644e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 30 17:38:23.571084 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:38:23.571042    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:38:24.420865 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:38:24.420831    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-m5rfb" podUID="418014eb-a790-40be-93c6-6ea289e644e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 30 17:38:34.420281 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:38:34.420242    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-m5rfb" podUID="418014eb-a790-40be-93c6-6ea289e644e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 30 17:38:44.420156 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:38:44.420114    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-m5rfb" podUID="418014eb-a790-40be-93c6-6ea289e644e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 30 17:38:54.420941 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:38:54.420902    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-m5rfb" podUID="418014eb-a790-40be-93c6-6ea289e644e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 30 17:39:04.420752 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:04.420710    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-m5rfb" podUID="418014eb-a790-40be-93c6-6ea289e644e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 30 17:39:14.421417 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:14.421388    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-m5rfb"
Jul 30 17:39:23.539068 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:23.538961    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-m5rfb"]
Jul 30 17:39:23.539595 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:23.539272    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-m5rfb" podUID="418014eb-a790-40be-93c6-6ea289e644e1" containerName="kserve-container" containerID="cri-o://01490904fef3e84e5b3840895251ba4d481990698261569d36eaccf22afd5781" gracePeriod=30
Jul 30 17:39:23.539595 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:23.539308    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-m5rfb" podUID="418014eb-a790-40be-93c6-6ea289e644e1" containerName="kube-rbac-proxy" containerID="cri-o://b19d314e1bbba5a305a2e35e5dbbe75d9c1a1015083bae1872739ab6c618c1cc" gracePeriod=30
Jul 30 17:39:23.613183 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:23.613148    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:39:23.813258 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:23.813227    2555 generic.go:356] "Generic (PLEG): container finished" podID="418014eb-a790-40be-93c6-6ea289e644e1" containerID="b19d314e1bbba5a305a2e35e5dbbe75d9c1a1015083bae1872739ab6c618c1cc" exitCode=2
Jul 30 17:39:23.813443 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:23.813301    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-m5rfb" event={"ID":"418014eb-a790-40be-93c6-6ea289e644e1","Type":"ContainerDied","Data":"b19d314e1bbba5a305a2e35e5dbbe75d9c1a1015083bae1872739ab6c618c1cc"}
Jul 30 17:39:24.414930 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:24.414893    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-m5rfb" podUID="418014eb-a790-40be-93c6-6ea289e644e1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.42:8643/healthz\": dial tcp 10.133.0.42:8643: connect: connection refused"
Jul 30 17:39:24.420314 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:24.420290    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-m5rfb" podUID="418014eb-a790-40be-93c6-6ea289e644e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 30 17:39:27.993294 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:27.993271    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-m5rfb"
Jul 30 17:39:28.089881 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:28.089806    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b6q4z\" (UniqueName: \"kubernetes.io/projected/418014eb-a790-40be-93c6-6ea289e644e1-kube-api-access-b6q4z\") pod \"418014eb-a790-40be-93c6-6ea289e644e1\" (UID: \"418014eb-a790-40be-93c6-6ea289e644e1\") "
Jul 30 17:39:28.089881 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:28.089847    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/418014eb-a790-40be-93c6-6ea289e644e1-kserve-provision-location\") pod \"418014eb-a790-40be-93c6-6ea289e644e1\" (UID: \"418014eb-a790-40be-93c6-6ea289e644e1\") "
Jul 30 17:39:28.089881 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:28.089870    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/418014eb-a790-40be-93c6-6ea289e644e1-kube-rbac-proxy-sar-config\") pod \"418014eb-a790-40be-93c6-6ea289e644e1\" (UID: \"418014eb-a790-40be-93c6-6ea289e644e1\") "
Jul 30 17:39:28.090183 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:28.089915    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/418014eb-a790-40be-93c6-6ea289e644e1-cabundle-cert\") pod \"418014eb-a790-40be-93c6-6ea289e644e1\" (UID: \"418014eb-a790-40be-93c6-6ea289e644e1\") "
Jul 30 17:39:28.090183 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:28.089973    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/418014eb-a790-40be-93c6-6ea289e644e1-proxy-tls\") pod \"418014eb-a790-40be-93c6-6ea289e644e1\" (UID: \"418014eb-a790-40be-93c6-6ea289e644e1\") "
Jul 30 17:39:28.090316 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:28.090217    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/418014eb-a790-40be-93c6-6ea289e644e1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "418014eb-a790-40be-93c6-6ea289e644e1" (UID: "418014eb-a790-40be-93c6-6ea289e644e1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 17:39:28.090316 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:28.090261    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/418014eb-a790-40be-93c6-6ea289e644e1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "418014eb-a790-40be-93c6-6ea289e644e1" (UID: "418014eb-a790-40be-93c6-6ea289e644e1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:39:28.090398 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:28.090309    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/418014eb-a790-40be-93c6-6ea289e644e1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "418014eb-a790-40be-93c6-6ea289e644e1" (UID: "418014eb-a790-40be-93c6-6ea289e644e1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:39:28.091999 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:28.091979    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/418014eb-a790-40be-93c6-6ea289e644e1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "418014eb-a790-40be-93c6-6ea289e644e1" (UID: "418014eb-a790-40be-93c6-6ea289e644e1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:39:28.092104 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:28.092042    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/418014eb-a790-40be-93c6-6ea289e644e1-kube-api-access-b6q4z" (OuterVolumeSpecName: "kube-api-access-b6q4z") pod "418014eb-a790-40be-93c6-6ea289e644e1" (UID: "418014eb-a790-40be-93c6-6ea289e644e1"). InnerVolumeSpecName "kube-api-access-b6q4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 17:39:28.190699 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:28.190664    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/418014eb-a790-40be-93c6-6ea289e644e1-proxy-tls\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:39:28.190699 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:28.190693    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6q4z\" (UniqueName: \"kubernetes.io/projected/418014eb-a790-40be-93c6-6ea289e644e1-kube-api-access-b6q4z\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:39:28.190699 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:28.190707    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/418014eb-a790-40be-93c6-6ea289e644e1-kserve-provision-location\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:39:28.190934 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:28.190719    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/418014eb-a790-40be-93c6-6ea289e644e1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:39:28.190934 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:28.190731    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/418014eb-a790-40be-93c6-6ea289e644e1-cabundle-cert\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:39:28.554690 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:28.554654    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:39:28.640464 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:28.640429    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-c2lc7"]
Jul 30 17:39:28.640810 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:28.640797    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="418014eb-a790-40be-93c6-6ea289e644e1" containerName="kube-rbac-proxy"
Jul 30 17:39:28.640866 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:28.640811    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="418014eb-a790-40be-93c6-6ea289e644e1" containerName="kube-rbac-proxy"
Jul 30 17:39:28.640866 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:28.640822    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f08cda5-90a1-431d-b6ae-85f6cf073099" containerName="kube-rbac-proxy"
Jul 30 17:39:28.640866 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:28.640829    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f08cda5-90a1-431d-b6ae-85f6cf073099" containerName="kube-rbac-proxy"
Jul 30 17:39:28.640866 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:28.640838    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f08cda5-90a1-431d-b6ae-85f6cf073099" containerName="storage-initializer"
Jul 30 17:39:28.640866 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:28.640844    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f08cda5-90a1-431d-b6ae-85f6cf073099" containerName="storage-initializer"
Jul 30 17:39:28.640866 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:28.640851    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f08cda5-90a1-431d-b6ae-85f6cf073099" containerName="kserve-container"
Jul 30 17:39:28.640866 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:28.640857    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f08cda5-90a1-431d-b6ae-85f6cf073099" containerName="kserve-container"
Jul 30 17:39:28.640866 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:28.640865    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="418014eb-a790-40be-93c6-6ea289e644e1" containerName="kserve-container"
Jul 30 17:39:28.640866 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:28.640870    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="418014eb-a790-40be-93c6-6ea289e644e1" containerName="kserve-container"
Jul 30 17:39:28.641170 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:28.640880    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="418014eb-a790-40be-93c6-6ea289e644e1" containerName="storage-initializer"
Jul 30 17:39:28.641170 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:28.640886    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="418014eb-a790-40be-93c6-6ea289e644e1" containerName="storage-initializer"
Jul 30 17:39:28.641170 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:28.640944    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f08cda5-90a1-431d-b6ae-85f6cf073099" containerName="kserve-container"
Jul 30 17:39:28.641170 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:28.640953    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="418014eb-a790-40be-93c6-6ea289e644e1" containerName="kserve-container"
Jul 30 17:39:28.641170 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:28.640962    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="418014eb-a790-40be-93c6-6ea289e644e1" containerName="kube-rbac-proxy"
Jul 30 17:39:28.641170 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:28.640968    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f08cda5-90a1-431d-b6ae-85f6cf073099" containerName="kube-rbac-proxy"
Jul 30 17:39:28.644593 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:28.644577    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-c2lc7"
Jul 30 17:39:28.647554 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:28.647531    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-predictor-serving-cert\""
Jul 30 17:39:28.647554 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:28.647547    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\""
Jul 30 17:39:28.656156 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:28.656130    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-c2lc7"]
Jul 30 17:39:28.796737 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:28.796687    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b0af284-1037-4024-9ed7-3e7ec59dc1bc-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-c2lc7\" (UID: \"5b0af284-1037-4024-9ed7-3e7ec59dc1bc\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-c2lc7"
Jul 30 17:39:28.796926 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:28.796813    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b0af284-1037-4024-9ed7-3e7ec59dc1bc-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-c2lc7\" (UID: \"5b0af284-1037-4024-9ed7-3e7ec59dc1bc\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-c2lc7"
Jul 30 17:39:28.796926 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:28.796873    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b0af284-1037-4024-9ed7-3e7ec59dc1bc-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-c2lc7\" (UID: \"5b0af284-1037-4024-9ed7-3e7ec59dc1bc\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-c2lc7"
Jul 30 17:39:28.796926 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:28.796903    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5b0af284-1037-4024-9ed7-3e7ec59dc1bc-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-c2lc7\" (UID: \"5b0af284-1037-4024-9ed7-3e7ec59dc1bc\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-c2lc7"
Jul 30 17:39:28.797073 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:28.796939    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcx9f\" (UniqueName: \"kubernetes.io/projected/5b0af284-1037-4024-9ed7-3e7ec59dc1bc-kube-api-access-bcx9f\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-c2lc7\" (UID: \"5b0af284-1037-4024-9ed7-3e7ec59dc1bc\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-c2lc7"
Jul 30 17:39:28.843157 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:28.843071    2555 generic.go:356] "Generic (PLEG): container finished" podID="418014eb-a790-40be-93c6-6ea289e644e1" containerID="01490904fef3e84e5b3840895251ba4d481990698261569d36eaccf22afd5781" exitCode=0
Jul 30 17:39:28.843157 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:28.843123    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-m5rfb" event={"ID":"418014eb-a790-40be-93c6-6ea289e644e1","Type":"ContainerDied","Data":"01490904fef3e84e5b3840895251ba4d481990698261569d36eaccf22afd5781"}
Jul 30 17:39:28.843157 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:28.843148    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-m5rfb"
Jul 30 17:39:28.843427 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:28.843211    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-m5rfb" event={"ID":"418014eb-a790-40be-93c6-6ea289e644e1","Type":"ContainerDied","Data":"bb8f273c3eb1c0b4891ff8647beffae8e154153dc62d42678400226c012ec462"}
Jul 30 17:39:28.843427 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:28.843281    2555 scope.go:117] "RemoveContainer" containerID="b19d314e1bbba5a305a2e35e5dbbe75d9c1a1015083bae1872739ab6c618c1cc"
Jul 30 17:39:28.851916 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:28.851896    2555 scope.go:117] "RemoveContainer" containerID="01490904fef3e84e5b3840895251ba4d481990698261569d36eaccf22afd5781"
Jul 30 17:39:28.859518 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:28.859323    2555 scope.go:117] "RemoveContainer" containerID="e6949377549660323cc1e670c4e5d3b96146d419ccd520f5a0ef7fa53a3d7c76"
Jul 30 17:39:28.866531 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:28.866514    2555 scope.go:117] "RemoveContainer" containerID="b19d314e1bbba5a305a2e35e5dbbe75d9c1a1015083bae1872739ab6c618c1cc"
Jul 30 17:39:28.866790 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:39:28.866771    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b19d314e1bbba5a305a2e35e5dbbe75d9c1a1015083bae1872739ab6c618c1cc\": container with ID starting with b19d314e1bbba5a305a2e35e5dbbe75d9c1a1015083bae1872739ab6c618c1cc not found: ID does not exist" containerID="b19d314e1bbba5a305a2e35e5dbbe75d9c1a1015083bae1872739ab6c618c1cc"
Jul 30 17:39:28.866866 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:28.866799    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b19d314e1bbba5a305a2e35e5dbbe75d9c1a1015083bae1872739ab6c618c1cc"} err="failed to get container status \"b19d314e1bbba5a305a2e35e5dbbe75d9c1a1015083bae1872739ab6c618c1cc\": rpc error: code = NotFound desc = could not find container \"b19d314e1bbba5a305a2e35e5dbbe75d9c1a1015083bae1872739ab6c618c1cc\": container with ID starting with b19d314e1bbba5a305a2e35e5dbbe75d9c1a1015083bae1872739ab6c618c1cc not found: ID does not exist"
Jul 30 17:39:28.866866 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:28.866829    2555 scope.go:117] "RemoveContainer" containerID="01490904fef3e84e5b3840895251ba4d481990698261569d36eaccf22afd5781"
Jul 30 17:39:28.867159 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:39:28.867130    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01490904fef3e84e5b3840895251ba4d481990698261569d36eaccf22afd5781\": container with ID starting with 01490904fef3e84e5b3840895251ba4d481990698261569d36eaccf22afd5781 not found: ID does not exist" containerID="01490904fef3e84e5b3840895251ba4d481990698261569d36eaccf22afd5781"
Jul 30 17:39:28.867258 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:28.867166    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01490904fef3e84e5b3840895251ba4d481990698261569d36eaccf22afd5781"} err="failed to get container status \"01490904fef3e84e5b3840895251ba4d481990698261569d36eaccf22afd5781\": rpc error: code = NotFound desc = could not find container \"01490904fef3e84e5b3840895251ba4d481990698261569d36eaccf22afd5781\": container with ID starting with 01490904fef3e84e5b3840895251ba4d481990698261569d36eaccf22afd5781 not found: ID does not exist"
Jul 30 17:39:28.867258 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:28.867184    2555 scope.go:117] "RemoveContainer" containerID="e6949377549660323cc1e670c4e5d3b96146d419ccd520f5a0ef7fa53a3d7c76"
Jul 30 17:39:28.867478 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:39:28.867458    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6949377549660323cc1e670c4e5d3b96146d419ccd520f5a0ef7fa53a3d7c76\": container with ID starting with e6949377549660323cc1e670c4e5d3b96146d419ccd520f5a0ef7fa53a3d7c76 not found: ID does not exist" containerID="e6949377549660323cc1e670c4e5d3b96146d419ccd520f5a0ef7fa53a3d7c76"
Jul 30 17:39:28.867547 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:28.867491    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6949377549660323cc1e670c4e5d3b96146d419ccd520f5a0ef7fa53a3d7c76"} err="failed to get container status \"e6949377549660323cc1e670c4e5d3b96146d419ccd520f5a0ef7fa53a3d7c76\": rpc error: code = NotFound desc = could not find container \"e6949377549660323cc1e670c4e5d3b96146d419ccd520f5a0ef7fa53a3d7c76\": container with ID starting with e6949377549660323cc1e670c4e5d3b96146d419ccd520f5a0ef7fa53a3d7c76 not found: ID does not exist"
Jul 30 17:39:28.870068 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:28.870046    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-m5rfb"]
Jul 30 17:39:28.874610 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:28.874590    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-m5rfb"]
Jul 30 17:39:28.898268 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:28.898243    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b0af284-1037-4024-9ed7-3e7ec59dc1bc-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-c2lc7\" (UID: \"5b0af284-1037-4024-9ed7-3e7ec59dc1bc\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-c2lc7"
Jul 30 17:39:28.898377 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:28.898306    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b0af284-1037-4024-9ed7-3e7ec59dc1bc-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-c2lc7\" (UID: \"5b0af284-1037-4024-9ed7-3e7ec59dc1bc\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-c2lc7"
Jul 30 17:39:28.898377 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:28.898331    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b0af284-1037-4024-9ed7-3e7ec59dc1bc-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-c2lc7\" (UID: \"5b0af284-1037-4024-9ed7-3e7ec59dc1bc\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-c2lc7"
Jul 30 17:39:28.898377 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:28.898354    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5b0af284-1037-4024-9ed7-3e7ec59dc1bc-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-c2lc7\" (UID: \"5b0af284-1037-4024-9ed7-3e7ec59dc1bc\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-c2lc7"
Jul 30 17:39:28.898377 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:28.898373    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcx9f\" (UniqueName: \"kubernetes.io/projected/5b0af284-1037-4024-9ed7-3e7ec59dc1bc-kube-api-access-bcx9f\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-c2lc7\" (UID: \"5b0af284-1037-4024-9ed7-3e7ec59dc1bc\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-c2lc7"
Jul 30 17:39:28.898594 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:39:28.898485    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-serving-cert: secret "isvc-lightgbm-v2-runtime-predictor-serving-cert" not found
Jul 30 17:39:28.898594 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:39:28.898541    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b0af284-1037-4024-9ed7-3e7ec59dc1bc-proxy-tls podName:5b0af284-1037-4024-9ed7-3e7ec59dc1bc nodeName:}" failed. No retries permitted until 2026-07-30 17:39:29.398523498 +0000 UTC m=+2343.885890880 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5b0af284-1037-4024-9ed7-3e7ec59dc1bc-proxy-tls") pod "isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-c2lc7" (UID: "5b0af284-1037-4024-9ed7-3e7ec59dc1bc") : secret "isvc-lightgbm-v2-runtime-predictor-serving-cert" not found
Jul 30 17:39:28.898952 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:28.898931    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b0af284-1037-4024-9ed7-3e7ec59dc1bc-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-c2lc7\" (UID: \"5b0af284-1037-4024-9ed7-3e7ec59dc1bc\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-c2lc7"
Jul 30 17:39:28.899040 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:28.898992    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b0af284-1037-4024-9ed7-3e7ec59dc1bc-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-c2lc7\" (UID: \"5b0af284-1037-4024-9ed7-3e7ec59dc1bc\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-c2lc7"
Jul 30 17:39:28.899097 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:28.899079    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5b0af284-1037-4024-9ed7-3e7ec59dc1bc-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-c2lc7\" (UID: \"5b0af284-1037-4024-9ed7-3e7ec59dc1bc\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-c2lc7"
Jul 30 17:39:28.909917 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:28.909899    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcx9f\" (UniqueName: \"kubernetes.io/projected/5b0af284-1037-4024-9ed7-3e7ec59dc1bc-kube-api-access-bcx9f\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-c2lc7\" (UID: \"5b0af284-1037-4024-9ed7-3e7ec59dc1bc\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-c2lc7"
Jul 30 17:39:29.216492 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:29.216405    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:39:29.402084 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:29.402042    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b0af284-1037-4024-9ed7-3e7ec59dc1bc-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-c2lc7\" (UID: \"5b0af284-1037-4024-9ed7-3e7ec59dc1bc\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-c2lc7"
Jul 30 17:39:29.404497 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:29.404476    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b0af284-1037-4024-9ed7-3e7ec59dc1bc-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-c2lc7\" (UID: \"5b0af284-1037-4024-9ed7-3e7ec59dc1bc\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-c2lc7"
Jul 30 17:39:29.555370 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:29.555342    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-c2lc7"
Jul 30 17:39:29.694246 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:29.694220    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-c2lc7"]
Jul 30 17:39:29.695919 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:39:29.695878    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b0af284_1037_4024_9ed7_3e7ec59dc1bc.slice/crio-f85fe038f454c307e2bee3459c036036eaab92923a21059164297f8dd9fbdf94 WatchSource:0}: Error finding container f85fe038f454c307e2bee3459c036036eaab92923a21059164297f8dd9fbdf94: Status 404 returned error can't find the container with id f85fe038f454c307e2bee3459c036036eaab92923a21059164297f8dd9fbdf94
Jul 30 17:39:29.850284 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:29.850196    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-c2lc7" event={"ID":"5b0af284-1037-4024-9ed7-3e7ec59dc1bc","Type":"ContainerStarted","Data":"45e5a71628ee45670f8361dc3c5ff432781eff40c6c1b5829a78e38a9ee7bf76"}
Jul 30 17:39:29.850284 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:29.850274    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-c2lc7" event={"ID":"5b0af284-1037-4024-9ed7-3e7ec59dc1bc","Type":"ContainerStarted","Data":"f85fe038f454c307e2bee3459c036036eaab92923a21059164297f8dd9fbdf94"}
Jul 30 17:39:30.061714 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:30.061681    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="418014eb-a790-40be-93c6-6ea289e644e1" path="/var/lib/kubelet/pods/418014eb-a790-40be-93c6-6ea289e644e1/volumes"
Jul 30 17:39:33.870406 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:33.870373    2555 generic.go:356] "Generic (PLEG): container finished" podID="5b0af284-1037-4024-9ed7-3e7ec59dc1bc" containerID="45e5a71628ee45670f8361dc3c5ff432781eff40c6c1b5829a78e38a9ee7bf76" exitCode=0
Jul 30 17:39:33.870900 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:39:33.870454    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-c2lc7" event={"ID":"5b0af284-1037-4024-9ed7-3e7ec59dc1bc","Type":"ContainerDied","Data":"45e5a71628ee45670f8361dc3c5ff432781eff40c6c1b5829a78e38a9ee7bf76"}
Jul 30 17:40:46.017971 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:40:46.017922    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:41:43.573461 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:41:43.573429    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-c2lc7" event={"ID":"5b0af284-1037-4024-9ed7-3e7ec59dc1bc","Type":"ContainerStarted","Data":"e9269094782513912b12683208a8e10e9c19c3f4e24cb4381b096f31effdc74f"}
Jul 30 17:41:43.574054 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:41:43.573487    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-c2lc7" event={"ID":"5b0af284-1037-4024-9ed7-3e7ec59dc1bc","Type":"ContainerStarted","Data":"62e81ed3a172213e81dbec23bd7daae6654478a4159135e0429bb89d836a07c8"}
Jul 30 17:41:43.574054 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:41:43.573584    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-c2lc7"
Jul 30 17:41:43.606289 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:41:43.606241    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-c2lc7" podStartSLOduration=2.429928382 podStartE2EDuration="2m15.606222793s" totalImagesPullingTime="2m9.514635274s" totalInitContainerRuntime="3.661659137s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:39:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:41:43.604757074 +0000 UTC m=+2478.092124487" watchObservedRunningTime="2026-07-30 17:41:43.606222793 +0000 UTC m=+2478.093590193"
Jul 30 17:41:44.579023 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:41:44.578976    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-c2lc7"
Jul 30 17:41:50.590610 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:41:50.590582    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-c2lc7"
Jul 30 17:41:56.703538 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:41:56.703507    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:42:20.594707 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:20.594636    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-c2lc7"
Jul 30 17:42:28.772136 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:28.772100    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-c2lc7"]
Jul 30 17:42:28.772665 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:28.772366    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-c2lc7" podUID="5b0af284-1037-4024-9ed7-3e7ec59dc1bc" containerName="kserve-container" containerID="cri-o://62e81ed3a172213e81dbec23bd7daae6654478a4159135e0429bb89d836a07c8" gracePeriod=30
Jul 30 17:42:28.772665 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:28.772403    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-c2lc7" podUID="5b0af284-1037-4024-9ed7-3e7ec59dc1bc" containerName="kube-rbac-proxy" containerID="cri-o://e9269094782513912b12683208a8e10e9c19c3f4e24cb4381b096f31effdc74f" gracePeriod=30
Jul 30 17:42:28.779816 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:28.779790    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:42:29.793334 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:29.793304    2555 generic.go:356] "Generic (PLEG): container finished" podID="5b0af284-1037-4024-9ed7-3e7ec59dc1bc" containerID="e9269094782513912b12683208a8e10e9c19c3f4e24cb4381b096f31effdc74f" exitCode=2
Jul 30 17:42:29.793334 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:29.793330    2555 generic.go:356] "Generic (PLEG): container finished" podID="5b0af284-1037-4024-9ed7-3e7ec59dc1bc" containerID="62e81ed3a172213e81dbec23bd7daae6654478a4159135e0429bb89d836a07c8" exitCode=0
Jul 30 17:42:29.793751 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:29.793374    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-c2lc7" event={"ID":"5b0af284-1037-4024-9ed7-3e7ec59dc1bc","Type":"ContainerDied","Data":"e9269094782513912b12683208a8e10e9c19c3f4e24cb4381b096f31effdc74f"}
Jul 30 17:42:29.793751 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:29.793453    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-c2lc7" event={"ID":"5b0af284-1037-4024-9ed7-3e7ec59dc1bc","Type":"ContainerDied","Data":"62e81ed3a172213e81dbec23bd7daae6654478a4159135e0429bb89d836a07c8"}
Jul 30 17:42:29.913837 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:29.913815    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-c2lc7"
Jul 30 17:42:29.989891 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:29.989862    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5b0af284-1037-4024-9ed7-3e7ec59dc1bc-cabundle-cert\") pod \"5b0af284-1037-4024-9ed7-3e7ec59dc1bc\" (UID: \"5b0af284-1037-4024-9ed7-3e7ec59dc1bc\") "
Jul 30 17:42:29.990097 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:29.989908    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b0af284-1037-4024-9ed7-3e7ec59dc1bc-kserve-provision-location\") pod \"5b0af284-1037-4024-9ed7-3e7ec59dc1bc\" (UID: \"5b0af284-1037-4024-9ed7-3e7ec59dc1bc\") "
Jul 30 17:42:29.990097 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:29.989929    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bcx9f\" (UniqueName: \"kubernetes.io/projected/5b0af284-1037-4024-9ed7-3e7ec59dc1bc-kube-api-access-bcx9f\") pod \"5b0af284-1037-4024-9ed7-3e7ec59dc1bc\" (UID: \"5b0af284-1037-4024-9ed7-3e7ec59dc1bc\") "
Jul 30 17:42:29.990097 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:29.989950    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b0af284-1037-4024-9ed7-3e7ec59dc1bc-kube-rbac-proxy-sar-config\") pod \"5b0af284-1037-4024-9ed7-3e7ec59dc1bc\" (UID: \"5b0af284-1037-4024-9ed7-3e7ec59dc1bc\") "
Jul 30 17:42:29.990097 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:29.990020    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b0af284-1037-4024-9ed7-3e7ec59dc1bc-proxy-tls\") pod \"5b0af284-1037-4024-9ed7-3e7ec59dc1bc\" (UID: \"5b0af284-1037-4024-9ed7-3e7ec59dc1bc\") "
Jul 30 17:42:29.990336 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:29.990277    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b0af284-1037-4024-9ed7-3e7ec59dc1bc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5b0af284-1037-4024-9ed7-3e7ec59dc1bc" (UID: "5b0af284-1037-4024-9ed7-3e7ec59dc1bc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 17:42:29.990388 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:29.990325    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b0af284-1037-4024-9ed7-3e7ec59dc1bc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5b0af284-1037-4024-9ed7-3e7ec59dc1bc" (UID: "5b0af284-1037-4024-9ed7-3e7ec59dc1bc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:42:29.990388 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:29.990331    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b0af284-1037-4024-9ed7-3e7ec59dc1bc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5b0af284-1037-4024-9ed7-3e7ec59dc1bc" (UID: "5b0af284-1037-4024-9ed7-3e7ec59dc1bc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:42:29.992055 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:29.992028    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b0af284-1037-4024-9ed7-3e7ec59dc1bc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5b0af284-1037-4024-9ed7-3e7ec59dc1bc" (UID: "5b0af284-1037-4024-9ed7-3e7ec59dc1bc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:42:29.992055 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:29.992037    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b0af284-1037-4024-9ed7-3e7ec59dc1bc-kube-api-access-bcx9f" (OuterVolumeSpecName: "kube-api-access-bcx9f") pod "5b0af284-1037-4024-9ed7-3e7ec59dc1bc" (UID: "5b0af284-1037-4024-9ed7-3e7ec59dc1bc"). InnerVolumeSpecName "kube-api-access-bcx9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 17:42:30.090898 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:30.090875    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b0af284-1037-4024-9ed7-3e7ec59dc1bc-proxy-tls\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:42:30.090898 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:30.090898    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5b0af284-1037-4024-9ed7-3e7ec59dc1bc-cabundle-cert\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:42:30.091080 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:30.090908    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b0af284-1037-4024-9ed7-3e7ec59dc1bc-kserve-provision-location\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:42:30.091080 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:30.090918    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcx9f\" (UniqueName: \"kubernetes.io/projected/5b0af284-1037-4024-9ed7-3e7ec59dc1bc-kube-api-access-bcx9f\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:42:30.091080 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:30.090927    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b0af284-1037-4024-9ed7-3e7ec59dc1bc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:42:30.799791 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:30.799766    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-c2lc7"
Jul 30 17:42:30.800235 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:30.799769    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-c2lc7" event={"ID":"5b0af284-1037-4024-9ed7-3e7ec59dc1bc","Type":"ContainerDied","Data":"f85fe038f454c307e2bee3459c036036eaab92923a21059164297f8dd9fbdf94"}
Jul 30 17:42:30.800235 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:30.799908    2555 scope.go:117] "RemoveContainer" containerID="e9269094782513912b12683208a8e10e9c19c3f4e24cb4381b096f31effdc74f"
Jul 30 17:42:30.808097 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:30.808076    2555 scope.go:117] "RemoveContainer" containerID="62e81ed3a172213e81dbec23bd7daae6654478a4159135e0429bb89d836a07c8"
Jul 30 17:42:30.815389 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:30.815371    2555 scope.go:117] "RemoveContainer" containerID="45e5a71628ee45670f8361dc3c5ff432781eff40c6c1b5829a78e38a9ee7bf76"
Jul 30 17:42:30.822520 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:30.822494    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-c2lc7"]
Jul 30 17:42:30.827347 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:30.827326    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-c2lc7"]
Jul 30 17:42:32.060640 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:32.060611    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b0af284-1037-4024-9ed7-3e7ec59dc1bc" path="/var/lib/kubelet/pods/5b0af284-1037-4024-9ed7-3e7ec59dc1bc/volumes"
Jul 30 17:42:33.843653 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:33.843627    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:42:33.918457 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:33.918424    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-fl5cg"]
Jul 30 17:42:33.918790 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:33.918777    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b0af284-1037-4024-9ed7-3e7ec59dc1bc" containerName="kserve-container"
Jul 30 17:42:33.918846 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:33.918791    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b0af284-1037-4024-9ed7-3e7ec59dc1bc" containerName="kserve-container"
Jul 30 17:42:33.918846 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:33.918807    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b0af284-1037-4024-9ed7-3e7ec59dc1bc" containerName="storage-initializer"
Jul 30 17:42:33.918846 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:33.918814    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b0af284-1037-4024-9ed7-3e7ec59dc1bc" containerName="storage-initializer"
Jul 30 17:42:33.918846 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:33.918823    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b0af284-1037-4024-9ed7-3e7ec59dc1bc" containerName="kube-rbac-proxy"
Jul 30 17:42:33.918846 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:33.918829    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b0af284-1037-4024-9ed7-3e7ec59dc1bc" containerName="kube-rbac-proxy"
Jul 30 17:42:33.919046 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:33.918887    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b0af284-1037-4024-9ed7-3e7ec59dc1bc" containerName="kube-rbac-proxy"
Jul 30 17:42:33.919046 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:33.918897    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b0af284-1037-4024-9ed7-3e7ec59dc1bc" containerName="kserve-container"
Jul 30 17:42:33.921193 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:33.921168    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-fl5cg"
Jul 30 17:42:33.923937 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:33.923916    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 30 17:42:33.923937 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:33.923930    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-predictor-serving-cert\""
Jul 30 17:42:33.924154 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:33.924034    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-plw5t\""
Jul 30 17:42:33.924154 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:33.924100    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 30 17:42:33.925085 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:33.925069    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 30 17:42:33.925154 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:33.925084    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 30 17:42:33.925154 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:33.925126    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\""
Jul 30 17:42:33.935849 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:33.935828    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-fl5cg"]
Jul 30 17:42:34.015997 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:34.015956    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/42a62eb7-c0cf-473d-ad9d-0233bfd05b09-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-fl5cg\" (UID: \"42a62eb7-c0cf-473d-ad9d-0233bfd05b09\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-fl5cg"
Jul 30 17:42:34.016214 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:34.016085    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nzcm\" (UniqueName: \"kubernetes.io/projected/42a62eb7-c0cf-473d-ad9d-0233bfd05b09-kube-api-access-6nzcm\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-fl5cg\" (UID: \"42a62eb7-c0cf-473d-ad9d-0233bfd05b09\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-fl5cg"
Jul 30 17:42:34.016214 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:34.016120    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42a62eb7-c0cf-473d-ad9d-0233bfd05b09-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-fl5cg\" (UID: \"42a62eb7-c0cf-473d-ad9d-0233bfd05b09\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-fl5cg"
Jul 30 17:42:34.016214 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:34.016168    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/42a62eb7-c0cf-473d-ad9d-0233bfd05b09-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-fl5cg\" (UID: \"42a62eb7-c0cf-473d-ad9d-0233bfd05b09\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-fl5cg"
Jul 30 17:42:34.016374 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:34.016249    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/42a62eb7-c0cf-473d-ad9d-0233bfd05b09-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-fl5cg\" (UID: \"42a62eb7-c0cf-473d-ad9d-0233bfd05b09\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-fl5cg"
Jul 30 17:42:34.117576 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:34.117471    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nzcm\" (UniqueName: \"kubernetes.io/projected/42a62eb7-c0cf-473d-ad9d-0233bfd05b09-kube-api-access-6nzcm\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-fl5cg\" (UID: \"42a62eb7-c0cf-473d-ad9d-0233bfd05b09\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-fl5cg"
Jul 30 17:42:34.117576 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:34.117530    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42a62eb7-c0cf-473d-ad9d-0233bfd05b09-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-fl5cg\" (UID: \"42a62eb7-c0cf-473d-ad9d-0233bfd05b09\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-fl5cg"
Jul 30 17:42:34.117576 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:34.117580    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/42a62eb7-c0cf-473d-ad9d-0233bfd05b09-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-fl5cg\" (UID: \"42a62eb7-c0cf-473d-ad9d-0233bfd05b09\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-fl5cg"
Jul 30 17:42:34.117869 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:34.117620    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/42a62eb7-c0cf-473d-ad9d-0233bfd05b09-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-fl5cg\" (UID: \"42a62eb7-c0cf-473d-ad9d-0233bfd05b09\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-fl5cg"
Jul 30 17:42:34.117869 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:34.117673    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/42a62eb7-c0cf-473d-ad9d-0233bfd05b09-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-fl5cg\" (UID: \"42a62eb7-c0cf-473d-ad9d-0233bfd05b09\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-fl5cg"
Jul 30 17:42:34.117869 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:42:34.117701    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-serving-cert: secret "isvc-lightgbm-v2-kserve-predictor-serving-cert" not found
Jul 30 17:42:34.117869 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:42:34.117775    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42a62eb7-c0cf-473d-ad9d-0233bfd05b09-proxy-tls podName:42a62eb7-c0cf-473d-ad9d-0233bfd05b09 nodeName:}" failed. No retries permitted until 2026-07-30 17:42:34.617753993 +0000 UTC m=+2529.105121372 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/42a62eb7-c0cf-473d-ad9d-0233bfd05b09-proxy-tls") pod "isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-fl5cg" (UID: "42a62eb7-c0cf-473d-ad9d-0233bfd05b09") : secret "isvc-lightgbm-v2-kserve-predictor-serving-cert" not found
Jul 30 17:42:34.118123 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:34.118100    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/42a62eb7-c0cf-473d-ad9d-0233bfd05b09-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-fl5cg\" (UID: \"42a62eb7-c0cf-473d-ad9d-0233bfd05b09\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-fl5cg"
Jul 30 17:42:34.118323 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:34.118303    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/42a62eb7-c0cf-473d-ad9d-0233bfd05b09-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-fl5cg\" (UID: \"42a62eb7-c0cf-473d-ad9d-0233bfd05b09\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-fl5cg"
Jul 30 17:42:34.118467 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:34.118447    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/42a62eb7-c0cf-473d-ad9d-0233bfd05b09-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-fl5cg\" (UID: \"42a62eb7-c0cf-473d-ad9d-0233bfd05b09\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-fl5cg"
Jul 30 17:42:34.127632 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:34.127605    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nzcm\" (UniqueName: \"kubernetes.io/projected/42a62eb7-c0cf-473d-ad9d-0233bfd05b09-kube-api-access-6nzcm\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-fl5cg\" (UID: \"42a62eb7-c0cf-473d-ad9d-0233bfd05b09\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-fl5cg"
Jul 30 17:42:34.204896 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:34.204861    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:42:34.621817 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:34.621772    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42a62eb7-c0cf-473d-ad9d-0233bfd05b09-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-fl5cg\" (UID: \"42a62eb7-c0cf-473d-ad9d-0233bfd05b09\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-fl5cg"
Jul 30 17:42:34.626623 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:34.626510    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42a62eb7-c0cf-473d-ad9d-0233bfd05b09-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-fl5cg\" (UID: \"42a62eb7-c0cf-473d-ad9d-0233bfd05b09\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-fl5cg"
Jul 30 17:42:34.832622 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:34.832590    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-fl5cg"
Jul 30 17:42:34.967065 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:34.967043    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-fl5cg"]
Jul 30 17:42:34.968735 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:42:34.968709    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42a62eb7_c0cf_473d_ad9d_0233bfd05b09.slice/crio-d68f615fd2d13ae272f4d4c827f719b8c619108972efa46f271b0ee02aa79a03 WatchSource:0}: Error finding container d68f615fd2d13ae272f4d4c827f719b8c619108972efa46f271b0ee02aa79a03: Status 404 returned error can't find the container with id d68f615fd2d13ae272f4d4c827f719b8c619108972efa46f271b0ee02aa79a03
Jul 30 17:42:35.823951 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:35.823908    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-fl5cg" event={"ID":"42a62eb7-c0cf-473d-ad9d-0233bfd05b09","Type":"ContainerStarted","Data":"b10d1c80d259fd77bae66844e3a6ce5cb2899800725cea4ae648288e336a64de"}
Jul 30 17:42:35.824182 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:35.823976    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-fl5cg" event={"ID":"42a62eb7-c0cf-473d-ad9d-0233bfd05b09","Type":"ContainerStarted","Data":"d68f615fd2d13ae272f4d4c827f719b8c619108972efa46f271b0ee02aa79a03"}
Jul 30 17:42:39.842217 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:39.842183    2555 generic.go:356] "Generic (PLEG): container finished" podID="42a62eb7-c0cf-473d-ad9d-0233bfd05b09" containerID="b10d1c80d259fd77bae66844e3a6ce5cb2899800725cea4ae648288e336a64de" exitCode=0
Jul 30 17:42:39.842694 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:39.842259    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-fl5cg" event={"ID":"42a62eb7-c0cf-473d-ad9d-0233bfd05b09","Type":"ContainerDied","Data":"b10d1c80d259fd77bae66844e3a6ce5cb2899800725cea4ae648288e336a64de"}
Jul 30 17:42:40.850732 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:40.850698    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-fl5cg" event={"ID":"42a62eb7-c0cf-473d-ad9d-0233bfd05b09","Type":"ContainerStarted","Data":"f2c51812116afaafe9c04052f321762d50d71d53e9121d14a730cce22457c15f"}
Jul 30 17:42:40.851351 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:40.850767    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-fl5cg" event={"ID":"42a62eb7-c0cf-473d-ad9d-0233bfd05b09","Type":"ContainerStarted","Data":"f4177ad5291e9b62dffcdf9fad733ba4c883dc949d1be36b6fc5fea5751877a8"}
Jul 30 17:42:40.851351 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:40.851114    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-fl5cg"
Jul 30 17:42:40.851351 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:40.851282    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-fl5cg"
Jul 30 17:42:40.852713 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:40.852679    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-fl5cg" podUID="42a62eb7-c0cf-473d-ad9d-0233bfd05b09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 30 17:42:40.875532 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:40.875489    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-fl5cg" podStartSLOduration=3.8799021270000003 podStartE2EDuration="7.875473219s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.995571092s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:42:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:42:40.87291331 +0000 UTC m=+2535.360280712" watchObservedRunningTime="2026-07-30 17:42:40.875473219 +0000 UTC m=+2535.362840621"
Jul 30 17:42:41.857493 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:41.857455    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-fl5cg" podUID="42a62eb7-c0cf-473d-ad9d-0233bfd05b09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 30 17:42:46.862689 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:46.862660    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-fl5cg"
Jul 30 17:42:46.863434 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:46.863406    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-fl5cg" podUID="42a62eb7-c0cf-473d-ad9d-0233bfd05b09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 30 17:42:56.864840 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:42:56.864809    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-fl5cg"
Jul 30 17:43:03.954084 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:03.954051    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-fl5cg"]
Jul 30 17:43:03.954612 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:03.954342    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-fl5cg" podUID="42a62eb7-c0cf-473d-ad9d-0233bfd05b09" containerName="kserve-container" containerID="cri-o://f4177ad5291e9b62dffcdf9fad733ba4c883dc949d1be36b6fc5fea5751877a8" gracePeriod=30
Jul 30 17:43:03.954612 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:03.954386    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-fl5cg" podUID="42a62eb7-c0cf-473d-ad9d-0233bfd05b09" containerName="kube-rbac-proxy" containerID="cri-o://f2c51812116afaafe9c04052f321762d50d71d53e9121d14a730cce22457c15f" gracePeriod=30
Jul 30 17:43:03.981021 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:03.980981    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:43:04.601239 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:04.601215    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-fl5cg"
Jul 30 17:43:04.694311 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:04.694220    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/42a62eb7-c0cf-473d-ad9d-0233bfd05b09-cabundle-cert\") pod \"42a62eb7-c0cf-473d-ad9d-0233bfd05b09\" (UID: \"42a62eb7-c0cf-473d-ad9d-0233bfd05b09\") "
Jul 30 17:43:04.694311 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:04.694260    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42a62eb7-c0cf-473d-ad9d-0233bfd05b09-proxy-tls\") pod \"42a62eb7-c0cf-473d-ad9d-0233bfd05b09\" (UID: \"42a62eb7-c0cf-473d-ad9d-0233bfd05b09\") "
Jul 30 17:43:04.694530 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:04.694313    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6nzcm\" (UniqueName: \"kubernetes.io/projected/42a62eb7-c0cf-473d-ad9d-0233bfd05b09-kube-api-access-6nzcm\") pod \"42a62eb7-c0cf-473d-ad9d-0233bfd05b09\" (UID: \"42a62eb7-c0cf-473d-ad9d-0233bfd05b09\") "
Jul 30 17:43:04.694530 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:04.694355    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/42a62eb7-c0cf-473d-ad9d-0233bfd05b09-kserve-provision-location\") pod \"42a62eb7-c0cf-473d-ad9d-0233bfd05b09\" (UID: \"42a62eb7-c0cf-473d-ad9d-0233bfd05b09\") "
Jul 30 17:43:04.694530 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:04.694394    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/42a62eb7-c0cf-473d-ad9d-0233bfd05b09-kube-rbac-proxy-sar-config\") pod \"42a62eb7-c0cf-473d-ad9d-0233bfd05b09\" (UID: \"42a62eb7-c0cf-473d-ad9d-0233bfd05b09\") "
Jul 30 17:43:04.694700 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:04.694672    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42a62eb7-c0cf-473d-ad9d-0233bfd05b09-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "42a62eb7-c0cf-473d-ad9d-0233bfd05b09" (UID: "42a62eb7-c0cf-473d-ad9d-0233bfd05b09"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 17:43:04.694760 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:04.694696    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42a62eb7-c0cf-473d-ad9d-0233bfd05b09-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "42a62eb7-c0cf-473d-ad9d-0233bfd05b09" (UID: "42a62eb7-c0cf-473d-ad9d-0233bfd05b09"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:43:04.694808 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:04.694771    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42a62eb7-c0cf-473d-ad9d-0233bfd05b09-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "42a62eb7-c0cf-473d-ad9d-0233bfd05b09" (UID: "42a62eb7-c0cf-473d-ad9d-0233bfd05b09"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:43:04.696440 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:04.696424    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42a62eb7-c0cf-473d-ad9d-0233bfd05b09-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "42a62eb7-c0cf-473d-ad9d-0233bfd05b09" (UID: "42a62eb7-c0cf-473d-ad9d-0233bfd05b09"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:43:04.696510 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:04.696444    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42a62eb7-c0cf-473d-ad9d-0233bfd05b09-kube-api-access-6nzcm" (OuterVolumeSpecName: "kube-api-access-6nzcm") pod "42a62eb7-c0cf-473d-ad9d-0233bfd05b09" (UID: "42a62eb7-c0cf-473d-ad9d-0233bfd05b09"). InnerVolumeSpecName "kube-api-access-6nzcm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 17:43:04.795037 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:04.794972    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/42a62eb7-c0cf-473d-ad9d-0233bfd05b09-cabundle-cert\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:43:04.795037 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:04.795022    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42a62eb7-c0cf-473d-ad9d-0233bfd05b09-proxy-tls\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:43:04.795037 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:04.795032    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nzcm\" (UniqueName: \"kubernetes.io/projected/42a62eb7-c0cf-473d-ad9d-0233bfd05b09-kube-api-access-6nzcm\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:43:04.795037 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:04.795041    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/42a62eb7-c0cf-473d-ad9d-0233bfd05b09-kserve-provision-location\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:43:04.795037 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:04.795050    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/42a62eb7-c0cf-473d-ad9d-0233bfd05b09-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:43:04.966673 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:04.966584    2555 generic.go:356] "Generic (PLEG): container finished" podID="42a62eb7-c0cf-473d-ad9d-0233bfd05b09" containerID="f2c51812116afaafe9c04052f321762d50d71d53e9121d14a730cce22457c15f" exitCode=2
Jul 30 17:43:04.966673 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:04.966614    2555 generic.go:356] "Generic (PLEG): container finished" podID="42a62eb7-c0cf-473d-ad9d-0233bfd05b09" containerID="f4177ad5291e9b62dffcdf9fad733ba4c883dc949d1be36b6fc5fea5751877a8" exitCode=0
Jul 30 17:43:04.966673 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:04.966649    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-fl5cg"
Jul 30 17:43:04.966673 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:04.966668    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-fl5cg" event={"ID":"42a62eb7-c0cf-473d-ad9d-0233bfd05b09","Type":"ContainerDied","Data":"f2c51812116afaafe9c04052f321762d50d71d53e9121d14a730cce22457c15f"}
Jul 30 17:43:04.967219 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:04.966726    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-fl5cg" event={"ID":"42a62eb7-c0cf-473d-ad9d-0233bfd05b09","Type":"ContainerDied","Data":"f4177ad5291e9b62dffcdf9fad733ba4c883dc949d1be36b6fc5fea5751877a8"}
Jul 30 17:43:04.967219 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:04.966746    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-fl5cg" event={"ID":"42a62eb7-c0cf-473d-ad9d-0233bfd05b09","Type":"ContainerDied","Data":"d68f615fd2d13ae272f4d4c827f719b8c619108972efa46f271b0ee02aa79a03"}
Jul 30 17:43:04.967219 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:04.966771    2555 scope.go:117] "RemoveContainer" containerID="f2c51812116afaafe9c04052f321762d50d71d53e9121d14a730cce22457c15f"
Jul 30 17:43:04.975248 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:04.975229    2555 scope.go:117] "RemoveContainer" containerID="f4177ad5291e9b62dffcdf9fad733ba4c883dc949d1be36b6fc5fea5751877a8"
Jul 30 17:43:04.982746 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:04.982728    2555 scope.go:117] "RemoveContainer" containerID="b10d1c80d259fd77bae66844e3a6ce5cb2899800725cea4ae648288e336a64de"
Jul 30 17:43:04.990068 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:04.990045    2555 scope.go:117] "RemoveContainer" containerID="f2c51812116afaafe9c04052f321762d50d71d53e9121d14a730cce22457c15f"
Jul 30 17:43:04.991331 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:43:04.990381    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2c51812116afaafe9c04052f321762d50d71d53e9121d14a730cce22457c15f\": container with ID starting with f2c51812116afaafe9c04052f321762d50d71d53e9121d14a730cce22457c15f not found: ID does not exist" containerID="f2c51812116afaafe9c04052f321762d50d71d53e9121d14a730cce22457c15f"
Jul 30 17:43:04.991331 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:04.990420    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2c51812116afaafe9c04052f321762d50d71d53e9121d14a730cce22457c15f"} err="failed to get container status \"f2c51812116afaafe9c04052f321762d50d71d53e9121d14a730cce22457c15f\": rpc error: code = NotFound desc = could not find container \"f2c51812116afaafe9c04052f321762d50d71d53e9121d14a730cce22457c15f\": container with ID starting with f2c51812116afaafe9c04052f321762d50d71d53e9121d14a730cce22457c15f not found: ID does not exist"
Jul 30 17:43:04.991331 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:04.990443    2555 scope.go:117] "RemoveContainer" containerID="f4177ad5291e9b62dffcdf9fad733ba4c883dc949d1be36b6fc5fea5751877a8"
Jul 30 17:43:04.991331 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:43:04.990728    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4177ad5291e9b62dffcdf9fad733ba4c883dc949d1be36b6fc5fea5751877a8\": container with ID starting with f4177ad5291e9b62dffcdf9fad733ba4c883dc949d1be36b6fc5fea5751877a8 not found: ID does not exist" containerID="f4177ad5291e9b62dffcdf9fad733ba4c883dc949d1be36b6fc5fea5751877a8"
Jul 30 17:43:04.991331 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:04.990755    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4177ad5291e9b62dffcdf9fad733ba4c883dc949d1be36b6fc5fea5751877a8"} err="failed to get container status \"f4177ad5291e9b62dffcdf9fad733ba4c883dc949d1be36b6fc5fea5751877a8\": rpc error: code = NotFound desc = could not find container \"f4177ad5291e9b62dffcdf9fad733ba4c883dc949d1be36b6fc5fea5751877a8\": container with ID starting with f4177ad5291e9b62dffcdf9fad733ba4c883dc949d1be36b6fc5fea5751877a8 not found: ID does not exist"
Jul 30 17:43:04.991331 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:04.990775    2555 scope.go:117] "RemoveContainer" containerID="b10d1c80d259fd77bae66844e3a6ce5cb2899800725cea4ae648288e336a64de"
Jul 30 17:43:04.991331 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:43:04.990995    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b10d1c80d259fd77bae66844e3a6ce5cb2899800725cea4ae648288e336a64de\": container with ID starting with b10d1c80d259fd77bae66844e3a6ce5cb2899800725cea4ae648288e336a64de not found: ID does not exist" containerID="b10d1c80d259fd77bae66844e3a6ce5cb2899800725cea4ae648288e336a64de"
Jul 30 17:43:04.991331 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:04.991132    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b10d1c80d259fd77bae66844e3a6ce5cb2899800725cea4ae648288e336a64de"} err="failed to get container status \"b10d1c80d259fd77bae66844e3a6ce5cb2899800725cea4ae648288e336a64de\": rpc error: code = NotFound desc = could not find container \"b10d1c80d259fd77bae66844e3a6ce5cb2899800725cea4ae648288e336a64de\": container with ID starting with b10d1c80d259fd77bae66844e3a6ce5cb2899800725cea4ae648288e336a64de not found: ID does not exist"
Jul 30 17:43:04.991331 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:04.991150    2555 scope.go:117] "RemoveContainer" containerID="f2c51812116afaafe9c04052f321762d50d71d53e9121d14a730cce22457c15f"
Jul 30 17:43:04.991840 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:04.991367    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2c51812116afaafe9c04052f321762d50d71d53e9121d14a730cce22457c15f"} err="failed to get container status \"f2c51812116afaafe9c04052f321762d50d71d53e9121d14a730cce22457c15f\": rpc error: code = NotFound desc = could not find container \"f2c51812116afaafe9c04052f321762d50d71d53e9121d14a730cce22457c15f\": container with ID starting with f2c51812116afaafe9c04052f321762d50d71d53e9121d14a730cce22457c15f not found: ID does not exist"
Jul 30 17:43:04.991840 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:04.991388    2555 scope.go:117] "RemoveContainer" containerID="f4177ad5291e9b62dffcdf9fad733ba4c883dc949d1be36b6fc5fea5751877a8"
Jul 30 17:43:04.991840 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:04.991689    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4177ad5291e9b62dffcdf9fad733ba4c883dc949d1be36b6fc5fea5751877a8"} err="failed to get container status \"f4177ad5291e9b62dffcdf9fad733ba4c883dc949d1be36b6fc5fea5751877a8\": rpc error: code = NotFound desc = could not find container \"f4177ad5291e9b62dffcdf9fad733ba4c883dc949d1be36b6fc5fea5751877a8\": container with ID starting with f4177ad5291e9b62dffcdf9fad733ba4c883dc949d1be36b6fc5fea5751877a8 not found: ID does not exist"
Jul 30 17:43:04.991840 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:04.991732    2555 scope.go:117] "RemoveContainer" containerID="b10d1c80d259fd77bae66844e3a6ce5cb2899800725cea4ae648288e336a64de"
Jul 30 17:43:04.992053 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:04.992031    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b10d1c80d259fd77bae66844e3a6ce5cb2899800725cea4ae648288e336a64de"} err="failed to get container status \"b10d1c80d259fd77bae66844e3a6ce5cb2899800725cea4ae648288e336a64de\": rpc error: code = NotFound desc = could not find container \"b10d1c80d259fd77bae66844e3a6ce5cb2899800725cea4ae648288e336a64de\": container with ID starting with b10d1c80d259fd77bae66844e3a6ce5cb2899800725cea4ae648288e336a64de not found: ID does not exist"
Jul 30 17:43:04.993155 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:04.993135    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-fl5cg"]
Jul 30 17:43:04.997173 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:04.997153    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-fl5cg"]
Jul 30 17:43:06.060371 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:06.060321    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42a62eb7-c0cf-473d-ad9d-0233bfd05b09" path="/var/lib/kubelet/pods/42a62eb7-c0cf-473d-ad9d-0233bfd05b09/volumes"
Jul 30 17:43:09.094163 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:09.094131    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:43:09.135390 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:09.135356    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-jrvsx"]
Jul 30 17:43:09.135722 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:09.135709    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42a62eb7-c0cf-473d-ad9d-0233bfd05b09" containerName="kserve-container"
Jul 30 17:43:09.135762 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:09.135723    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="42a62eb7-c0cf-473d-ad9d-0233bfd05b09" containerName="kserve-container"
Jul 30 17:43:09.135762 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:09.135740    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42a62eb7-c0cf-473d-ad9d-0233bfd05b09" containerName="storage-initializer"
Jul 30 17:43:09.135762 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:09.135745    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="42a62eb7-c0cf-473d-ad9d-0233bfd05b09" containerName="storage-initializer"
Jul 30 17:43:09.135762 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:09.135753    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42a62eb7-c0cf-473d-ad9d-0233bfd05b09" containerName="kube-rbac-proxy"
Jul 30 17:43:09.135762 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:09.135758    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="42a62eb7-c0cf-473d-ad9d-0233bfd05b09" containerName="kube-rbac-proxy"
Jul 30 17:43:09.135917 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:09.135814    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="42a62eb7-c0cf-473d-ad9d-0233bfd05b09" containerName="kube-rbac-proxy"
Jul 30 17:43:09.135917 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:09.135824    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="42a62eb7-c0cf-473d-ad9d-0233bfd05b09" containerName="kserve-container"
Jul 30 17:43:09.139271 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:09.139252    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-jrvsx"
Jul 30 17:43:09.142138 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:09.142042    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 30 17:43:09.142138 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:09.142069    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\""
Jul 30 17:43:09.142138 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:09.142128    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 30 17:43:09.142394 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:09.142137    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 30 17:43:09.143384 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:09.143369    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 30 17:43:09.143446 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:09.143372    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-plw5t\""
Jul 30 17:43:09.143446 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:09.143372    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-predictor-serving-cert\""
Jul 30 17:43:09.150761 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:09.150739    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-jrvsx"]
Jul 30 17:43:09.239619 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:09.239585    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-jrvsx\" (UID: \"a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-jrvsx"
Jul 30 17:43:09.239789 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:09.239639    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-jrvsx\" (UID: \"a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-jrvsx"
Jul 30 17:43:09.239789 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:09.239708    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-jrvsx\" (UID: \"a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-jrvsx"
Jul 30 17:43:09.239789 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:09.239736    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-jrvsx\" (UID: \"a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-jrvsx"
Jul 30 17:43:09.239789 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:09.239774    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9xv6\" (UniqueName: \"kubernetes.io/projected/a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152-kube-api-access-b9xv6\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-jrvsx\" (UID: \"a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-jrvsx"
Jul 30 17:43:09.340728 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:09.340698    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-jrvsx\" (UID: \"a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-jrvsx"
Jul 30 17:43:09.340853 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:09.340733    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-jrvsx\" (UID: \"a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-jrvsx"
Jul 30 17:43:09.340853 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:09.340763    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9xv6\" (UniqueName: \"kubernetes.io/projected/a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152-kube-api-access-b9xv6\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-jrvsx\" (UID: \"a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-jrvsx"
Jul 30 17:43:09.340853 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:09.340808    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-jrvsx\" (UID: \"a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-jrvsx"
Jul 30 17:43:09.340978 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:09.340850    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-jrvsx\" (UID: \"a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-jrvsx"
Jul 30 17:43:09.340978 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:43:09.340870    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-serving-cert: secret "isvc-mlflow-v2-runtime-predictor-serving-cert" not found
Jul 30 17:43:09.340978 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:43:09.340940    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152-proxy-tls podName:a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152 nodeName:}" failed. No retries permitted until 2026-07-30 17:43:09.840918907 +0000 UTC m=+2564.328286291 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152-proxy-tls") pod "isvc-mlflow-v2-runtime-predictor-6456f94494-jrvsx" (UID: "a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152") : secret "isvc-mlflow-v2-runtime-predictor-serving-cert" not found
Jul 30 17:43:09.341308 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:09.341291    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-jrvsx\" (UID: \"a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-jrvsx"
Jul 30 17:43:09.341496 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:09.341475    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-jrvsx\" (UID: \"a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-jrvsx"
Jul 30 17:43:09.341537 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:09.341506    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-jrvsx\" (UID: \"a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-jrvsx"
Jul 30 17:43:09.350775 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:09.350727    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9xv6\" (UniqueName: \"kubernetes.io/projected/a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152-kube-api-access-b9xv6\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-jrvsx\" (UID: \"a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-jrvsx"
Jul 30 17:43:09.531920 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:09.531877    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:43:09.845376 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:09.845345    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-jrvsx\" (UID: \"a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-jrvsx"
Jul 30 17:43:09.847787 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:09.847762    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-jrvsx\" (UID: \"a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-jrvsx"
Jul 30 17:43:10.050872 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:10.050839    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-jrvsx"
Jul 30 17:43:10.182562 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:10.182536    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-jrvsx"]
Jul 30 17:43:10.184507 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:43:10.184474    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1ef3bb3_8fe7_4e3b_bdc7_d8627660d152.slice/crio-6699d2b86a09cccd6f3479dc49cd9aeafffb85ae84e08001e28b2e05eb90486f WatchSource:0}: Error finding container 6699d2b86a09cccd6f3479dc49cd9aeafffb85ae84e08001e28b2e05eb90486f: Status 404 returned error can't find the container with id 6699d2b86a09cccd6f3479dc49cd9aeafffb85ae84e08001e28b2e05eb90486f
Jul 30 17:43:11.003427 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:11.003395    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-jrvsx" event={"ID":"a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152","Type":"ContainerStarted","Data":"da61f21fafb15cf23cccd0eadddfbb0af392cf9ecd5b3eac565cc719791f5339"}
Jul 30 17:43:11.003617 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:11.003478    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-jrvsx" event={"ID":"a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152","Type":"ContainerStarted","Data":"6699d2b86a09cccd6f3479dc49cd9aeafffb85ae84e08001e28b2e05eb90486f"}
Jul 30 17:43:15.024592 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:15.024555    2555 generic.go:356] "Generic (PLEG): container finished" podID="a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152" containerID="da61f21fafb15cf23cccd0eadddfbb0af392cf9ecd5b3eac565cc719791f5339" exitCode=0
Jul 30 17:43:15.024977 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:15.024605    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-jrvsx" event={"ID":"a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152","Type":"ContainerDied","Data":"da61f21fafb15cf23cccd0eadddfbb0af392cf9ecd5b3eac565cc719791f5339"}
Jul 30 17:43:16.032287 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:16.032246    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-jrvsx" event={"ID":"a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152","Type":"ContainerStarted","Data":"dffb3fb02096bff2cc26062eac612252410b9ce0f2409a1248ee6f09466b1872"}
Jul 30 17:43:16.032660 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:16.032329    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-jrvsx" event={"ID":"a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152","Type":"ContainerStarted","Data":"ef1e8772899ad08527feb8a8b6a59e597b4e0f06e605191e6b33e4f2b2cab553"}
Jul 30 17:43:16.032660 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:16.032544    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-jrvsx"
Jul 30 17:43:16.055782 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:16.055736    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-jrvsx" podStartSLOduration=3.064855705 podStartE2EDuration="7.055721151s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.990865446s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:43:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:43:16.053914668 +0000 UTC m=+2570.541282070" watchObservedRunningTime="2026-07-30 17:43:16.055721151 +0000 UTC m=+2570.543088613"
Jul 30 17:43:17.037838 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:17.037807    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-jrvsx"
Jul 30 17:43:23.049028 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:23.048987    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-jrvsx"
Jul 30 17:43:53.053156 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:53.053067    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-jrvsx"
Jul 30 17:43:59.217323 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:59.217290    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-jrvsx"]
Jul 30 17:43:59.217832 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:59.217563    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-jrvsx" podUID="a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152" containerName="kserve-container" containerID="cri-o://ef1e8772899ad08527feb8a8b6a59e597b4e0f06e605191e6b33e4f2b2cab553" gracePeriod=30
Jul 30 17:43:59.217832 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:59.217616    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-jrvsx" podUID="a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152" containerName="kube-rbac-proxy" containerID="cri-o://dffb3fb02096bff2cc26062eac612252410b9ce0f2409a1248ee6f09466b1872" gracePeriod=30
Jul 30 17:43:59.239870 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:43:59.239846    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:44:00.240708 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:00.240630    2555 generic.go:356] "Generic (PLEG): container finished" podID="a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152" containerID="dffb3fb02096bff2cc26062eac612252410b9ce0f2409a1248ee6f09466b1872" exitCode=2
Jul 30 17:44:00.241176 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:00.240748    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-jrvsx" event={"ID":"a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152","Type":"ContainerDied","Data":"dffb3fb02096bff2cc26062eac612252410b9ce0f2409a1248ee6f09466b1872"}
Jul 30 17:44:00.387091 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:00.387066    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-jrvsx"
Jul 30 17:44:00.501602 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:00.501511    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b9xv6\" (UniqueName: \"kubernetes.io/projected/a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152-kube-api-access-b9xv6\") pod \"a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152\" (UID: \"a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152\") "
Jul 30 17:44:00.501602 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:00.501556    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152-proxy-tls\") pod \"a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152\" (UID: \"a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152\") "
Jul 30 17:44:00.501602 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:00.501577    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152-cabundle-cert\") pod \"a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152\" (UID: \"a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152\") "
Jul 30 17:44:00.501883 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:00.501612    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152-kube-rbac-proxy-sar-config\") pod \"a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152\" (UID: \"a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152\") "
Jul 30 17:44:00.501883 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:00.501723    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152-kserve-provision-location\") pod \"a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152\" (UID: \"a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152\") "
Jul 30 17:44:00.501996 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:00.501975    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152" (UID: "a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:44:00.502096 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:00.502035    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152" (UID: "a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:44:00.502145 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:00.502090    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152" (UID: "a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 17:44:00.503732 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:00.503704    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152" (UID: "a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:44:00.503846 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:00.503787    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152-kube-api-access-b9xv6" (OuterVolumeSpecName: "kube-api-access-b9xv6") pod "a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152" (UID: "a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152"). InnerVolumeSpecName "kube-api-access-b9xv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 17:44:00.603429 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:00.603393    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9xv6\" (UniqueName: \"kubernetes.io/projected/a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152-kube-api-access-b9xv6\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:44:00.603429 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:00.603423    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152-proxy-tls\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:44:00.603429 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:00.603432    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152-cabundle-cert\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:44:00.603663 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:00.603441    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:44:00.603663 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:00.603450    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152-kserve-provision-location\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:44:01.247239 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:01.247204    2555 generic.go:356] "Generic (PLEG): container finished" podID="a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152" containerID="ef1e8772899ad08527feb8a8b6a59e597b4e0f06e605191e6b33e4f2b2cab553" exitCode=0
Jul 30 17:44:01.247690 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:01.247264    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-jrvsx" event={"ID":"a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152","Type":"ContainerDied","Data":"ef1e8772899ad08527feb8a8b6a59e597b4e0f06e605191e6b33e4f2b2cab553"}
Jul 30 17:44:01.247690 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:01.247294    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-jrvsx"
Jul 30 17:44:01.247690 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:01.247310    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-jrvsx" event={"ID":"a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152","Type":"ContainerDied","Data":"6699d2b86a09cccd6f3479dc49cd9aeafffb85ae84e08001e28b2e05eb90486f"}
Jul 30 17:44:01.247690 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:01.247337    2555 scope.go:117] "RemoveContainer" containerID="dffb3fb02096bff2cc26062eac612252410b9ce0f2409a1248ee6f09466b1872"
Jul 30 17:44:01.256358 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:01.256128    2555 scope.go:117] "RemoveContainer" containerID="ef1e8772899ad08527feb8a8b6a59e597b4e0f06e605191e6b33e4f2b2cab553"
Jul 30 17:44:01.263977 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:01.263961    2555 scope.go:117] "RemoveContainer" containerID="da61f21fafb15cf23cccd0eadddfbb0af392cf9ecd5b3eac565cc719791f5339"
Jul 30 17:44:01.271673 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:01.271645    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-jrvsx"]
Jul 30 17:44:01.272535 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:01.272513    2555 scope.go:117] "RemoveContainer" containerID="dffb3fb02096bff2cc26062eac612252410b9ce0f2409a1248ee6f09466b1872"
Jul 30 17:44:01.272800 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:44:01.272784    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dffb3fb02096bff2cc26062eac612252410b9ce0f2409a1248ee6f09466b1872\": container with ID starting with dffb3fb02096bff2cc26062eac612252410b9ce0f2409a1248ee6f09466b1872 not found: ID does not exist" containerID="dffb3fb02096bff2cc26062eac612252410b9ce0f2409a1248ee6f09466b1872"
Jul 30 17:44:01.272846 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:01.272808    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dffb3fb02096bff2cc26062eac612252410b9ce0f2409a1248ee6f09466b1872"} err="failed to get container status \"dffb3fb02096bff2cc26062eac612252410b9ce0f2409a1248ee6f09466b1872\": rpc error: code = NotFound desc = could not find container \"dffb3fb02096bff2cc26062eac612252410b9ce0f2409a1248ee6f09466b1872\": container with ID starting with dffb3fb02096bff2cc26062eac612252410b9ce0f2409a1248ee6f09466b1872 not found: ID does not exist"
Jul 30 17:44:01.272846 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:01.272823    2555 scope.go:117] "RemoveContainer" containerID="ef1e8772899ad08527feb8a8b6a59e597b4e0f06e605191e6b33e4f2b2cab553"
Jul 30 17:44:01.273107 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:44:01.273083    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef1e8772899ad08527feb8a8b6a59e597b4e0f06e605191e6b33e4f2b2cab553\": container with ID starting with ef1e8772899ad08527feb8a8b6a59e597b4e0f06e605191e6b33e4f2b2cab553 not found: ID does not exist" containerID="ef1e8772899ad08527feb8a8b6a59e597b4e0f06e605191e6b33e4f2b2cab553"
Jul 30 17:44:01.273215 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:01.273111    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef1e8772899ad08527feb8a8b6a59e597b4e0f06e605191e6b33e4f2b2cab553"} err="failed to get container status \"ef1e8772899ad08527feb8a8b6a59e597b4e0f06e605191e6b33e4f2b2cab553\": rpc error: code = NotFound desc = could not find container \"ef1e8772899ad08527feb8a8b6a59e597b4e0f06e605191e6b33e4f2b2cab553\": container with ID starting with ef1e8772899ad08527feb8a8b6a59e597b4e0f06e605191e6b33e4f2b2cab553 not found: ID does not exist"
Jul 30 17:44:01.273215 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:01.273128    2555 scope.go:117] "RemoveContainer" containerID="da61f21fafb15cf23cccd0eadddfbb0af392cf9ecd5b3eac565cc719791f5339"
Jul 30 17:44:01.273442 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:44:01.273414    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da61f21fafb15cf23cccd0eadddfbb0af392cf9ecd5b3eac565cc719791f5339\": container with ID starting with da61f21fafb15cf23cccd0eadddfbb0af392cf9ecd5b3eac565cc719791f5339 not found: ID does not exist" containerID="da61f21fafb15cf23cccd0eadddfbb0af392cf9ecd5b3eac565cc719791f5339"
Jul 30 17:44:01.273576 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:01.273447    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da61f21fafb15cf23cccd0eadddfbb0af392cf9ecd5b3eac565cc719791f5339"} err="failed to get container status \"da61f21fafb15cf23cccd0eadddfbb0af392cf9ecd5b3eac565cc719791f5339\": rpc error: code = NotFound desc = could not find container \"da61f21fafb15cf23cccd0eadddfbb0af392cf9ecd5b3eac565cc719791f5339\": container with ID starting with da61f21fafb15cf23cccd0eadddfbb0af392cf9ecd5b3eac565cc719791f5339 not found: ID does not exist"
Jul 30 17:44:01.275675 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:01.275653    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-jrvsx"]
Jul 30 17:44:02.064270 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:02.062400    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152" path="/var/lib/kubelet/pods/a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152/volumes"
Jul 30 17:44:04.347489 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:04.347381    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:44:04.387431 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:04.387400    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c7d4b6846-6q7hx"]
Jul 30 17:44:04.387814 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:04.387799    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152" containerName="kube-rbac-proxy"
Jul 30 17:44:04.387871 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:04.387816    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152" containerName="kube-rbac-proxy"
Jul 30 17:44:04.387871 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:04.387832    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152" containerName="kserve-container"
Jul 30 17:44:04.387871 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:04.387838    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152" containerName="kserve-container"
Jul 30 17:44:04.387871 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:04.387846    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152" containerName="storage-initializer"
Jul 30 17:44:04.387871 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:04.387852    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152" containerName="storage-initializer"
Jul 30 17:44:04.388048 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:04.387918    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152" containerName="kserve-container"
Jul 30 17:44:04.388048 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:04.387930    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1ef3bb3-8fe7-4e3b-bdc7-d8627660d152" containerName="kube-rbac-proxy"
Jul 30 17:44:04.390658 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:04.390636    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c7d4b6846-6q7hx"
Jul 30 17:44:04.393932 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:04.393910    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 30 17:44:04.393932 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:04.393925    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-plw5t\""
Jul 30 17:44:04.394864 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:04.394843    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\""
Jul 30 17:44:04.394999 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:04.394861    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-predictor-serving-cert\""
Jul 30 17:44:04.394999 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:04.394925    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 30 17:44:04.394999 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:04.394933    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 30 17:44:04.398444 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:04.398328    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 30 17:44:04.412424 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:04.412396    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c7d4b6846-6q7hx"]
Jul 30 17:44:04.436729 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:04.436703    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c33fe433-6098-4aa5-9fa0-8d0451d1eb3b-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-6c7d4b6846-6q7hx\" (UID: \"c33fe433-6098-4aa5-9fa0-8d0451d1eb3b\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c7d4b6846-6q7hx"
Jul 30 17:44:04.436860 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:04.436785    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c33fe433-6098-4aa5-9fa0-8d0451d1eb3b-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-6c7d4b6846-6q7hx\" (UID: \"c33fe433-6098-4aa5-9fa0-8d0451d1eb3b\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c7d4b6846-6q7hx"
Jul 30 17:44:04.436860 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:04.436806    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjq5d\" (UniqueName: \"kubernetes.io/projected/c33fe433-6098-4aa5-9fa0-8d0451d1eb3b-kube-api-access-mjq5d\") pod \"isvc-sklearn-mcp-predictor-6c7d4b6846-6q7hx\" (UID: \"c33fe433-6098-4aa5-9fa0-8d0451d1eb3b\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c7d4b6846-6q7hx"
Jul 30 17:44:04.436860 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:04.436840    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c33fe433-6098-4aa5-9fa0-8d0451d1eb3b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-6c7d4b6846-6q7hx\" (UID: \"c33fe433-6098-4aa5-9fa0-8d0451d1eb3b\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c7d4b6846-6q7hx"
Jul 30 17:44:04.436964 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:04.436896    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c33fe433-6098-4aa5-9fa0-8d0451d1eb3b-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-6c7d4b6846-6q7hx\" (UID: \"c33fe433-6098-4aa5-9fa0-8d0451d1eb3b\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c7d4b6846-6q7hx"
Jul 30 17:44:04.496695 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:04.496666    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:44:04.537863 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:04.537832    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c33fe433-6098-4aa5-9fa0-8d0451d1eb3b-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-6c7d4b6846-6q7hx\" (UID: \"c33fe433-6098-4aa5-9fa0-8d0451d1eb3b\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c7d4b6846-6q7hx"
Jul 30 17:44:04.538031 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:04.537876    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c33fe433-6098-4aa5-9fa0-8d0451d1eb3b-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-6c7d4b6846-6q7hx\" (UID: \"c33fe433-6098-4aa5-9fa0-8d0451d1eb3b\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c7d4b6846-6q7hx"
Jul 30 17:44:04.538031 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:04.537930    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c33fe433-6098-4aa5-9fa0-8d0451d1eb3b-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-6c7d4b6846-6q7hx\" (UID: \"c33fe433-6098-4aa5-9fa0-8d0451d1eb3b\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c7d4b6846-6q7hx"
Jul 30 17:44:04.538031 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:04.537945    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjq5d\" (UniqueName: \"kubernetes.io/projected/c33fe433-6098-4aa5-9fa0-8d0451d1eb3b-kube-api-access-mjq5d\") pod \"isvc-sklearn-mcp-predictor-6c7d4b6846-6q7hx\" (UID: \"c33fe433-6098-4aa5-9fa0-8d0451d1eb3b\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c7d4b6846-6q7hx"
Jul 30 17:44:04.538031 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:04.537975    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c33fe433-6098-4aa5-9fa0-8d0451d1eb3b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-6c7d4b6846-6q7hx\" (UID: \"c33fe433-6098-4aa5-9fa0-8d0451d1eb3b\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c7d4b6846-6q7hx"
Jul 30 17:44:04.538328 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:04.538307    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c33fe433-6098-4aa5-9fa0-8d0451d1eb3b-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-6c7d4b6846-6q7hx\" (UID: \"c33fe433-6098-4aa5-9fa0-8d0451d1eb3b\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c7d4b6846-6q7hx"
Jul 30 17:44:04.538621 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:04.538594    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c33fe433-6098-4aa5-9fa0-8d0451d1eb3b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-6c7d4b6846-6q7hx\" (UID: \"c33fe433-6098-4aa5-9fa0-8d0451d1eb3b\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c7d4b6846-6q7hx"
Jul 30 17:44:04.538737 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:04.538634    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c33fe433-6098-4aa5-9fa0-8d0451d1eb3b-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-6c7d4b6846-6q7hx\" (UID: \"c33fe433-6098-4aa5-9fa0-8d0451d1eb3b\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c7d4b6846-6q7hx"
Jul 30 17:44:04.540503 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:04.540479    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c33fe433-6098-4aa5-9fa0-8d0451d1eb3b-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-6c7d4b6846-6q7hx\" (UID: \"c33fe433-6098-4aa5-9fa0-8d0451d1eb3b\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c7d4b6846-6q7hx"
Jul 30 17:44:04.548551 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:04.548519    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjq5d\" (UniqueName: \"kubernetes.io/projected/c33fe433-6098-4aa5-9fa0-8d0451d1eb3b-kube-api-access-mjq5d\") pod \"isvc-sklearn-mcp-predictor-6c7d4b6846-6q7hx\" (UID: \"c33fe433-6098-4aa5-9fa0-8d0451d1eb3b\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c7d4b6846-6q7hx"
Jul 30 17:44:04.701299 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:04.701190    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c7d4b6846-6q7hx"
Jul 30 17:44:04.866252 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:04.866227    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c7d4b6846-6q7hx"]
Jul 30 17:44:04.869053 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:44:04.869024    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc33fe433_6098_4aa5_9fa0_8d0451d1eb3b.slice/crio-8e517087f7102f4bb00a6736082e489c490a332499dfa926690da6f8757be723 WatchSource:0}: Error finding container 8e517087f7102f4bb00a6736082e489c490a332499dfa926690da6f8757be723: Status 404 returned error can't find the container with id 8e517087f7102f4bb00a6736082e489c490a332499dfa926690da6f8757be723
Jul 30 17:44:05.269052 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:05.269022    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c7d4b6846-6q7hx" event={"ID":"c33fe433-6098-4aa5-9fa0-8d0451d1eb3b","Type":"ContainerStarted","Data":"2f56ef195e7a79c3eee1116da7832aa3767e5c775e69ee8bd37b00567539584d"}
Jul 30 17:44:05.269234 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:05.269081    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c7d4b6846-6q7hx" event={"ID":"c33fe433-6098-4aa5-9fa0-8d0451d1eb3b","Type":"ContainerStarted","Data":"8e517087f7102f4bb00a6736082e489c490a332499dfa926690da6f8757be723"}
Jul 30 17:44:09.287953 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:09.287916    2555 generic.go:356] "Generic (PLEG): container finished" podID="c33fe433-6098-4aa5-9fa0-8d0451d1eb3b" containerID="2f56ef195e7a79c3eee1116da7832aa3767e5c775e69ee8bd37b00567539584d" exitCode=0
Jul 30 17:44:09.288467 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:09.287968    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c7d4b6846-6q7hx" event={"ID":"c33fe433-6098-4aa5-9fa0-8d0451d1eb3b","Type":"ContainerDied","Data":"2f56ef195e7a79c3eee1116da7832aa3767e5c775e69ee8bd37b00567539584d"}
Jul 30 17:44:09.348806 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:09.348785    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 30 17:44:10.295343 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:10.295307    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c7d4b6846-6q7hx" event={"ID":"c33fe433-6098-4aa5-9fa0-8d0451d1eb3b","Type":"ContainerStarted","Data":"d597ff0e1ba6022b6da8bd7c0063574c25ebe8c8514823b7e3f7b32a1f5297fd"}
Jul 30 17:44:12.310565 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:12.310528    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c7d4b6846-6q7hx" event={"ID":"c33fe433-6098-4aa5-9fa0-8d0451d1eb3b","Type":"ContainerStarted","Data":"eec8948e4fcc5a8a1ab9197e77c161d3cff1c22b8415dcd5a166811289b042df"}
Jul 30 17:44:12.311053 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:12.310609    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c7d4b6846-6q7hx" event={"ID":"c33fe433-6098-4aa5-9fa0-8d0451d1eb3b","Type":"ContainerStarted","Data":"a7bf54c135587a4048da29347b0531185a705e7189b319b4696d7f1138bb1618"}
Jul 30 17:44:12.311053 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:12.310860    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c7d4b6846-6q7hx"
Jul 30 17:44:12.311053 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:12.310923    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c7d4b6846-6q7hx"
Jul 30 17:44:12.311053 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:12.310960    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c7d4b6846-6q7hx"
Jul 30 17:44:12.336243 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:12.336202    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c7d4b6846-6q7hx" podStartSLOduration=1.785009348 podStartE2EDuration="8.336186133s" totalImagesPullingTime="2.742999183s" totalInitContainerRuntime="3.808177602s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:44:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:44:12.334956765 +0000 UTC m=+2626.822324175" watchObservedRunningTime="2026-07-30 17:44:12.336186133 +0000 UTC m=+2626.823553533"
Jul 30 17:44:18.329957 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:18.329929    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c7d4b6846-6q7hx"
Jul 30 17:44:44.702143 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:44.702100    2555 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c7d4b6846-6q7hx" podUID="c33fe433-6098-4aa5-9fa0-8d0451d1eb3b" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.46:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 30 17:44:48.334657 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:44:48.334619    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c7d4b6846-6q7hx"
Jul 30 17:45:18.335641 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:45:18.335569    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c7d4b6846-6q7hx"
Jul 30 17:45:24.525578 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:45:24.525542    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c7d4b6846-6q7hx"]
Jul 30 17:45:24.526070 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:45:24.525856    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c7d4b6846-6q7hx" podUID="c33fe433-6098-4aa5-9fa0-8d0451d1eb3b" containerName="kserve-container" containerID="cri-o://d597ff0e1ba6022b6da8bd7c0063574c25ebe8c8514823b7e3f7b32a1f5297fd" gracePeriod=30
Jul 30 17:45:24.526070 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:45:24.525911    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c7d4b6846-6q7hx" podUID="c33fe433-6098-4aa5-9fa0-8d0451d1eb3b" containerName="kube-rbac-proxy" containerID="cri-o://eec8948e4fcc5a8a1ab9197e77c161d3cff1c22b8415dcd5a166811289b042df" gracePeriod=30
Jul 30 17:45:24.526070 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:45:24.525969    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c7d4b6846-6q7hx" podUID="c33fe433-6098-4aa5-9fa0-8d0451d1eb3b" containerName="kserve-agent" containerID="cri-o://a7bf54c135587a4048da29347b0531185a705e7189b319b4696d7f1138bb1618" gracePeriod=30
Jul 30 17:45:24.556383 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:45:24.556352    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:45:24.651919 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:45:24.651893    2555 generic.go:356] "Generic (PLEG): container finished" podID="c33fe433-6098-4aa5-9fa0-8d0451d1eb3b" containerID="eec8948e4fcc5a8a1ab9197e77c161d3cff1c22b8415dcd5a166811289b042df" exitCode=2
Jul 30 17:45:24.652151 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:45:24.651968    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c7d4b6846-6q7hx" event={"ID":"c33fe433-6098-4aa5-9fa0-8d0451d1eb3b","Type":"ContainerDied","Data":"eec8948e4fcc5a8a1ab9197e77c161d3cff1c22b8415dcd5a166811289b042df"}
Jul 30 17:45:27.667033 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:45:27.666981    2555 generic.go:356] "Generic (PLEG): container finished" podID="c33fe433-6098-4aa5-9fa0-8d0451d1eb3b" containerID="d597ff0e1ba6022b6da8bd7c0063574c25ebe8c8514823b7e3f7b32a1f5297fd" exitCode=0
Jul 30 17:45:27.667418 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:45:27.667045    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c7d4b6846-6q7hx" event={"ID":"c33fe433-6098-4aa5-9fa0-8d0451d1eb3b","Type":"ContainerDied","Data":"d597ff0e1ba6022b6da8bd7c0063574c25ebe8c8514823b7e3f7b32a1f5297fd"}
Jul 30 17:45:28.324488 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:45:28.324448    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c7d4b6846-6q7hx" podUID="c33fe433-6098-4aa5-9fa0-8d0451d1eb3b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.46:8643/healthz\": dial tcp 10.133.0.46:8643: connect: connection refused"
Jul 30 17:45:28.330958 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:45:28.330931    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c7d4b6846-6q7hx" podUID="c33fe433-6098-4aa5-9fa0-8d0451d1eb3b" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.46:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 30 17:45:29.646648 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:45:29.646604    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:45:30.463658 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:45:30.463628    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:45:33.324597 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:45:33.324547    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c7d4b6846-6q7hx" podUID="c33fe433-6098-4aa5-9fa0-8d0451d1eb3b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.46:8643/healthz\": dial tcp 10.133.0.46:8643: connect: connection refused"
Jul 30 17:45:38.324414 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:45:38.324370    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c7d4b6846-6q7hx" podUID="c33fe433-6098-4aa5-9fa0-8d0451d1eb3b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.46:8643/healthz\": dial tcp 10.133.0.46:8643: connect: connection refused"
Jul 30 17:45:38.324893 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:45:38.324514    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c7d4b6846-6q7hx"
Jul 30 17:45:38.331094 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:45:38.331063    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c7d4b6846-6q7hx" podUID="c33fe433-6098-4aa5-9fa0-8d0451d1eb3b" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.46:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 30 17:45:43.324347 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:45:43.324304    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c7d4b6846-6q7hx" podUID="c33fe433-6098-4aa5-9fa0-8d0451d1eb3b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.46:8643/healthz\": dial tcp 10.133.0.46:8643: connect: connection refused"
Jul 30 17:45:48.324904 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:45:48.324861    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c7d4b6846-6q7hx" podUID="c33fe433-6098-4aa5-9fa0-8d0451d1eb3b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.46:8643/healthz\": dial tcp 10.133.0.46:8643: connect: connection refused"
Jul 30 17:45:48.331288 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:45:48.331265    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c7d4b6846-6q7hx" podUID="c33fe433-6098-4aa5-9fa0-8d0451d1eb3b" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.46:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 30 17:45:48.331375 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:45:48.331363    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c7d4b6846-6q7hx"
Jul 30 17:45:53.324793 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:45:53.324752    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c7d4b6846-6q7hx" podUID="c33fe433-6098-4aa5-9fa0-8d0451d1eb3b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.46:8643/healthz\": dial tcp 10.133.0.46:8643: connect: connection refused"
Jul 30 17:45:53.470281 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:45:53.470240    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:45:54.724269 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:45:54.724246    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c7d4b6846-6q7hx"
Jul 30 17:45:54.795121 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:45:54.795093    2555 generic.go:356] "Generic (PLEG): container finished" podID="c33fe433-6098-4aa5-9fa0-8d0451d1eb3b" containerID="a7bf54c135587a4048da29347b0531185a705e7189b319b4696d7f1138bb1618" exitCode=0
Jul 30 17:45:54.795279 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:45:54.795158    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c7d4b6846-6q7hx"
Jul 30 17:45:54.795279 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:45:54.795154    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c7d4b6846-6q7hx" event={"ID":"c33fe433-6098-4aa5-9fa0-8d0451d1eb3b","Type":"ContainerDied","Data":"a7bf54c135587a4048da29347b0531185a705e7189b319b4696d7f1138bb1618"}
Jul 30 17:45:54.795279 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:45:54.795235    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c7d4b6846-6q7hx" event={"ID":"c33fe433-6098-4aa5-9fa0-8d0451d1eb3b","Type":"ContainerDied","Data":"8e517087f7102f4bb00a6736082e489c490a332499dfa926690da6f8757be723"}
Jul 30 17:45:54.795388 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:45:54.795281    2555 scope.go:117] "RemoveContainer" containerID="eec8948e4fcc5a8a1ab9197e77c161d3cff1c22b8415dcd5a166811289b042df"
Jul 30 17:45:54.803605 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:45:54.803590    2555 scope.go:117] "RemoveContainer" containerID="a7bf54c135587a4048da29347b0531185a705e7189b319b4696d7f1138bb1618"
Jul 30 17:45:54.810818 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:45:54.810800    2555 scope.go:117] "RemoveContainer" containerID="d597ff0e1ba6022b6da8bd7c0063574c25ebe8c8514823b7e3f7b32a1f5297fd"
Jul 30 17:45:54.811810 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:45:54.811786    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c33fe433-6098-4aa5-9fa0-8d0451d1eb3b-cabundle-cert\") pod \"c33fe433-6098-4aa5-9fa0-8d0451d1eb3b\" (UID: \"c33fe433-6098-4aa5-9fa0-8d0451d1eb3b\") "
Jul 30 17:45:54.811869 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:45:54.811823    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c33fe433-6098-4aa5-9fa0-8d0451d1eb3b-proxy-tls\") pod \"c33fe433-6098-4aa5-9fa0-8d0451d1eb3b\" (UID: \"c33fe433-6098-4aa5-9fa0-8d0451d1eb3b\") "
Jul 30 17:45:54.811869 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:45:54.811851    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c33fe433-6098-4aa5-9fa0-8d0451d1eb3b-kserve-provision-location\") pod \"c33fe433-6098-4aa5-9fa0-8d0451d1eb3b\" (UID: \"c33fe433-6098-4aa5-9fa0-8d0451d1eb3b\") "
Jul 30 17:45:54.812075 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:45:54.811900    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c33fe433-6098-4aa5-9fa0-8d0451d1eb3b-kube-rbac-proxy-sar-config\") pod \"c33fe433-6098-4aa5-9fa0-8d0451d1eb3b\" (UID: \"c33fe433-6098-4aa5-9fa0-8d0451d1eb3b\") "
Jul 30 17:45:54.812075 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:45:54.811929    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mjq5d\" (UniqueName: \"kubernetes.io/projected/c33fe433-6098-4aa5-9fa0-8d0451d1eb3b-kube-api-access-mjq5d\") pod \"c33fe433-6098-4aa5-9fa0-8d0451d1eb3b\" (UID: \"c33fe433-6098-4aa5-9fa0-8d0451d1eb3b\") "
Jul 30 17:45:54.812199 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:45:54.812174    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c33fe433-6098-4aa5-9fa0-8d0451d1eb3b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c33fe433-6098-4aa5-9fa0-8d0451d1eb3b" (UID: "c33fe433-6098-4aa5-9fa0-8d0451d1eb3b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 17:45:54.812399 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:45:54.812370    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c33fe433-6098-4aa5-9fa0-8d0451d1eb3b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c33fe433-6098-4aa5-9fa0-8d0451d1eb3b" (UID: "c33fe433-6098-4aa5-9fa0-8d0451d1eb3b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:45:54.812498 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:45:54.812481    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c33fe433-6098-4aa5-9fa0-8d0451d1eb3b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c33fe433-6098-4aa5-9fa0-8d0451d1eb3b" (UID: "c33fe433-6098-4aa5-9fa0-8d0451d1eb3b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:45:54.814156 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:45:54.814128    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c33fe433-6098-4aa5-9fa0-8d0451d1eb3b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c33fe433-6098-4aa5-9fa0-8d0451d1eb3b" (UID: "c33fe433-6098-4aa5-9fa0-8d0451d1eb3b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:45:54.814156 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:45:54.814135    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c33fe433-6098-4aa5-9fa0-8d0451d1eb3b-kube-api-access-mjq5d" (OuterVolumeSpecName: "kube-api-access-mjq5d") pod "c33fe433-6098-4aa5-9fa0-8d0451d1eb3b" (UID: "c33fe433-6098-4aa5-9fa0-8d0451d1eb3b"). InnerVolumeSpecName "kube-api-access-mjq5d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 17:45:54.829437 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:45:54.829417    2555 scope.go:117] "RemoveContainer" containerID="2f56ef195e7a79c3eee1116da7832aa3767e5c775e69ee8bd37b00567539584d"
Jul 30 17:45:54.836808 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:45:54.836792    2555 scope.go:117] "RemoveContainer" containerID="eec8948e4fcc5a8a1ab9197e77c161d3cff1c22b8415dcd5a166811289b042df"
Jul 30 17:45:54.837060 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:45:54.837041    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eec8948e4fcc5a8a1ab9197e77c161d3cff1c22b8415dcd5a166811289b042df\": container with ID starting with eec8948e4fcc5a8a1ab9197e77c161d3cff1c22b8415dcd5a166811289b042df not found: ID does not exist" containerID="eec8948e4fcc5a8a1ab9197e77c161d3cff1c22b8415dcd5a166811289b042df"
Jul 30 17:45:54.837123 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:45:54.837069    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eec8948e4fcc5a8a1ab9197e77c161d3cff1c22b8415dcd5a166811289b042df"} err="failed to get container status \"eec8948e4fcc5a8a1ab9197e77c161d3cff1c22b8415dcd5a166811289b042df\": rpc error: code = NotFound desc = could not find container \"eec8948e4fcc5a8a1ab9197e77c161d3cff1c22b8415dcd5a166811289b042df\": container with ID starting with eec8948e4fcc5a8a1ab9197e77c161d3cff1c22b8415dcd5a166811289b042df not found: ID does not exist"
Jul 30 17:45:54.837123 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:45:54.837086    2555 scope.go:117] "RemoveContainer" containerID="a7bf54c135587a4048da29347b0531185a705e7189b319b4696d7f1138bb1618"
Jul 30 17:45:54.837296 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:45:54.837279    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7bf54c135587a4048da29347b0531185a705e7189b319b4696d7f1138bb1618\": container with ID starting with a7bf54c135587a4048da29347b0531185a705e7189b319b4696d7f1138bb1618 not found: ID does not exist" containerID="a7bf54c135587a4048da29347b0531185a705e7189b319b4696d7f1138bb1618"
Jul 30 17:45:54.837345 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:45:54.837300    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7bf54c135587a4048da29347b0531185a705e7189b319b4696d7f1138bb1618"} err="failed to get container status \"a7bf54c135587a4048da29347b0531185a705e7189b319b4696d7f1138bb1618\": rpc error: code = NotFound desc = could not find container \"a7bf54c135587a4048da29347b0531185a705e7189b319b4696d7f1138bb1618\": container with ID starting with a7bf54c135587a4048da29347b0531185a705e7189b319b4696d7f1138bb1618 not found: ID does not exist"
Jul 30 17:45:54.837345 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:45:54.837311    2555 scope.go:117] "RemoveContainer" containerID="d597ff0e1ba6022b6da8bd7c0063574c25ebe8c8514823b7e3f7b32a1f5297fd"
Jul 30 17:45:54.837535 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:45:54.837519    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d597ff0e1ba6022b6da8bd7c0063574c25ebe8c8514823b7e3f7b32a1f5297fd\": container with ID starting with d597ff0e1ba6022b6da8bd7c0063574c25ebe8c8514823b7e3f7b32a1f5297fd not found: ID does not exist" containerID="d597ff0e1ba6022b6da8bd7c0063574c25ebe8c8514823b7e3f7b32a1f5297fd"
Jul 30 17:45:54.837598 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:45:54.837541    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d597ff0e1ba6022b6da8bd7c0063574c25ebe8c8514823b7e3f7b32a1f5297fd"} err="failed to get container status \"d597ff0e1ba6022b6da8bd7c0063574c25ebe8c8514823b7e3f7b32a1f5297fd\": rpc error: code = NotFound desc = could not find container \"d597ff0e1ba6022b6da8bd7c0063574c25ebe8c8514823b7e3f7b32a1f5297fd\": container with ID starting with d597ff0e1ba6022b6da8bd7c0063574c25ebe8c8514823b7e3f7b32a1f5297fd not found: ID does not exist"
Jul 30 17:45:54.837598 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:45:54.837556    2555 scope.go:117] "RemoveContainer" containerID="2f56ef195e7a79c3eee1116da7832aa3767e5c775e69ee8bd37b00567539584d"
Jul 30 17:45:54.837770 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:45:54.837748    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f56ef195e7a79c3eee1116da7832aa3767e5c775e69ee8bd37b00567539584d\": container with ID starting with 2f56ef195e7a79c3eee1116da7832aa3767e5c775e69ee8bd37b00567539584d not found: ID does not exist" containerID="2f56ef195e7a79c3eee1116da7832aa3767e5c775e69ee8bd37b00567539584d"
Jul 30 17:45:54.837873 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:45:54.837772    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f56ef195e7a79c3eee1116da7832aa3767e5c775e69ee8bd37b00567539584d"} err="failed to get container status \"2f56ef195e7a79c3eee1116da7832aa3767e5c775e69ee8bd37b00567539584d\": rpc error: code = NotFound desc = could not find container \"2f56ef195e7a79c3eee1116da7832aa3767e5c775e69ee8bd37b00567539584d\": container with ID starting with 2f56ef195e7a79c3eee1116da7832aa3767e5c775e69ee8bd37b00567539584d not found: ID does not exist"
Jul 30 17:45:54.912888 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:45:54.912854    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c33fe433-6098-4aa5-9fa0-8d0451d1eb3b-cabundle-cert\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:45:54.912888 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:45:54.912885    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c33fe433-6098-4aa5-9fa0-8d0451d1eb3b-proxy-tls\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:45:54.913117 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:45:54.912895    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c33fe433-6098-4aa5-9fa0-8d0451d1eb3b-kserve-provision-location\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:45:54.913117 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:45:54.912911    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c33fe433-6098-4aa5-9fa0-8d0451d1eb3b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:45:54.913117 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:45:54.912922    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjq5d\" (UniqueName: \"kubernetes.io/projected/c33fe433-6098-4aa5-9fa0-8d0451d1eb3b-kube-api-access-mjq5d\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:45:55.122376 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:45:55.122342    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c7d4b6846-6q7hx"]
Jul 30 17:45:55.133660 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:45:55.128886    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c7d4b6846-6q7hx"]
Jul 30 17:45:56.061178 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:45:56.061147    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c33fe433-6098-4aa5-9fa0-8d0451d1eb3b" path="/var/lib/kubelet/pods/c33fe433-6098-4aa5-9fa0-8d0451d1eb3b/volumes"
Jul 30 17:46:41.315874 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:46:41.315827    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:46:46.467381 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:46:46.467346    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:46:46.501226 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:46:46.501193    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kv6ls"]
Jul 30 17:46:46.501569 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:46:46.501557    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c33fe433-6098-4aa5-9fa0-8d0451d1eb3b" containerName="kube-rbac-proxy"
Jul 30 17:46:46.501618 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:46:46.501571    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c33fe433-6098-4aa5-9fa0-8d0451d1eb3b" containerName="kube-rbac-proxy"
Jul 30 17:46:46.501618 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:46:46.501590    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c33fe433-6098-4aa5-9fa0-8d0451d1eb3b" containerName="storage-initializer"
Jul 30 17:46:46.501618 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:46:46.501596    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c33fe433-6098-4aa5-9fa0-8d0451d1eb3b" containerName="storage-initializer"
Jul 30 17:46:46.501618 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:46:46.501607    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c33fe433-6098-4aa5-9fa0-8d0451d1eb3b" containerName="kserve-container"
Jul 30 17:46:46.501618 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:46:46.501612    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c33fe433-6098-4aa5-9fa0-8d0451d1eb3b" containerName="kserve-container"
Jul 30 17:46:46.501793 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:46:46.501620    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c33fe433-6098-4aa5-9fa0-8d0451d1eb3b" containerName="kserve-agent"
Jul 30 17:46:46.501793 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:46:46.501626    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c33fe433-6098-4aa5-9fa0-8d0451d1eb3b" containerName="kserve-agent"
Jul 30 17:46:46.501793 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:46:46.501676    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c33fe433-6098-4aa5-9fa0-8d0451d1eb3b" containerName="kserve-container"
Jul 30 17:46:46.501793 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:46:46.501683    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c33fe433-6098-4aa5-9fa0-8d0451d1eb3b" containerName="kserve-agent"
Jul 30 17:46:46.501793 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:46:46.501693    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c33fe433-6098-4aa5-9fa0-8d0451d1eb3b" containerName="kube-rbac-proxy"
Jul 30 17:46:46.504919 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:46:46.504900    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kv6ls"
Jul 30 17:46:46.510018 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:46:46.509983    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-kube-rbac-proxy-sar-config\""
Jul 30 17:46:46.510131 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:46:46.510032    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 30 17:46:46.510131 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:46:46.510051    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-predictor-serving-cert\""
Jul 30 17:46:46.510356 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:46:46.510343    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 30 17:46:46.510429 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:46:46.510371    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 30 17:46:46.511163 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:46:46.511145    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 30 17:46:46.511266 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:46:46.511151    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-plw5t\""
Jul 30 17:46:46.538182 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:46:46.538154    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kv6ls"]
Jul 30 17:46:46.565453 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:46:46.565424    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8654c17b-d90b-4b18-929b-cde4eae70e15-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-kv6ls\" (UID: \"8654c17b-d90b-4b18-929b-cde4eae70e15\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kv6ls"
Jul 30 17:46:46.565623 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:46:46.565463    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8654c17b-d90b-4b18-929b-cde4eae70e15-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-kv6ls\" (UID: \"8654c17b-d90b-4b18-929b-cde4eae70e15\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kv6ls"
Jul 30 17:46:46.565623 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:46:46.565493    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8654c17b-d90b-4b18-929b-cde4eae70e15-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-kv6ls\" (UID: \"8654c17b-d90b-4b18-929b-cde4eae70e15\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kv6ls"
Jul 30 17:46:46.565623 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:46:46.565577    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8654c17b-d90b-4b18-929b-cde4eae70e15-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-kv6ls\" (UID: \"8654c17b-d90b-4b18-929b-cde4eae70e15\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kv6ls"
Jul 30 17:46:46.565735 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:46:46.565636    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdhmd\" (UniqueName: \"kubernetes.io/projected/8654c17b-d90b-4b18-929b-cde4eae70e15-kube-api-access-jdhmd\") pod \"isvc-paddle-runtime-predictor-58b99fff76-kv6ls\" (UID: \"8654c17b-d90b-4b18-929b-cde4eae70e15\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kv6ls"
Jul 30 17:46:46.616129 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:46:46.616092    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:46:46.666291 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:46:46.666252    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8654c17b-d90b-4b18-929b-cde4eae70e15-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-kv6ls\" (UID: \"8654c17b-d90b-4b18-929b-cde4eae70e15\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kv6ls"
Jul 30 17:46:46.666291 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:46:46.666293    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8654c17b-d90b-4b18-929b-cde4eae70e15-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-kv6ls\" (UID: \"8654c17b-d90b-4b18-929b-cde4eae70e15\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kv6ls"
Jul 30 17:46:46.666541 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:46:46.666413    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8654c17b-d90b-4b18-929b-cde4eae70e15-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-kv6ls\" (UID: \"8654c17b-d90b-4b18-929b-cde4eae70e15\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kv6ls"
Jul 30 17:46:46.666541 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:46:46.666500    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8654c17b-d90b-4b18-929b-cde4eae70e15-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-kv6ls\" (UID: \"8654c17b-d90b-4b18-929b-cde4eae70e15\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kv6ls"
Jul 30 17:46:46.666541 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:46:46.666536    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdhmd\" (UniqueName: \"kubernetes.io/projected/8654c17b-d90b-4b18-929b-cde4eae70e15-kube-api-access-jdhmd\") pod \"isvc-paddle-runtime-predictor-58b99fff76-kv6ls\" (UID: \"8654c17b-d90b-4b18-929b-cde4eae70e15\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kv6ls"
Jul 30 17:46:46.666956 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:46:46.666930    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8654c17b-d90b-4b18-929b-cde4eae70e15-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-kv6ls\" (UID: \"8654c17b-d90b-4b18-929b-cde4eae70e15\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kv6ls"
Jul 30 17:46:46.667086 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:46:46.666941    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8654c17b-d90b-4b18-929b-cde4eae70e15-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-kv6ls\" (UID: \"8654c17b-d90b-4b18-929b-cde4eae70e15\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kv6ls"
Jul 30 17:46:46.667086 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:46:46.667070    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8654c17b-d90b-4b18-929b-cde4eae70e15-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-kv6ls\" (UID: \"8654c17b-d90b-4b18-929b-cde4eae70e15\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kv6ls"
Jul 30 17:46:46.668905 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:46:46.668884    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8654c17b-d90b-4b18-929b-cde4eae70e15-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-kv6ls\" (UID: \"8654c17b-d90b-4b18-929b-cde4eae70e15\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kv6ls"
Jul 30 17:46:46.678376 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:46:46.678349    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdhmd\" (UniqueName: \"kubernetes.io/projected/8654c17b-d90b-4b18-929b-cde4eae70e15-kube-api-access-jdhmd\") pod \"isvc-paddle-runtime-predictor-58b99fff76-kv6ls\" (UID: \"8654c17b-d90b-4b18-929b-cde4eae70e15\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kv6ls"
Jul 30 17:46:46.816162 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:46:46.816131    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kv6ls"
Jul 30 17:46:46.982883 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:46:46.982858    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kv6ls"]
Jul 30 17:46:46.983758 ip-10-0-143-190 kubenswrapper[2555]: W0730 17:46:46.983735    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8654c17b_d90b_4b18_929b_cde4eae70e15.slice/crio-ef39f1c9245054269bb409930ffdd84b3a20fc425f2b7aacfdb7b999a5253d93 WatchSource:0}: Error finding container ef39f1c9245054269bb409930ffdd84b3a20fc425f2b7aacfdb7b999a5253d93: Status 404 returned error can't find the container with id ef39f1c9245054269bb409930ffdd84b3a20fc425f2b7aacfdb7b999a5253d93
Jul 30 17:46:47.033663 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:46:47.033636    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kv6ls" event={"ID":"8654c17b-d90b-4b18-929b-cde4eae70e15","Type":"ContainerStarted","Data":"ef39f1c9245054269bb409930ffdd84b3a20fc425f2b7aacfdb7b999a5253d93"}
Jul 30 17:46:48.039575 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:46:48.039489    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kv6ls" event={"ID":"8654c17b-d90b-4b18-929b-cde4eae70e15","Type":"ContainerStarted","Data":"c43fe11ac04dc935f659ccfb23a42cfd0c15dc1fc32e43236522d7b20797d4ae"}
Jul 30 17:46:52.059804 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:46:52.059771    2555 generic.go:356] "Generic (PLEG): container finished" podID="8654c17b-d90b-4b18-929b-cde4eae70e15" containerID="c43fe11ac04dc935f659ccfb23a42cfd0c15dc1fc32e43236522d7b20797d4ae" exitCode=0
Jul 30 17:46:52.060204 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:46:52.059949    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kv6ls" event={"ID":"8654c17b-d90b-4b18-929b-cde4eae70e15","Type":"ContainerDied","Data":"c43fe11ac04dc935f659ccfb23a42cfd0c15dc1fc32e43236522d7b20797d4ae"}
Jul 30 17:47:04.126326 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:47:04.126292    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kv6ls" event={"ID":"8654c17b-d90b-4b18-929b-cde4eae70e15","Type":"ContainerStarted","Data":"87062daf21f9abe6df62504a56488a2b81689e4e7b3f7af8c96c38bd5e724693"}
Jul 30 17:47:04.126756 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:47:04.126354    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kv6ls" event={"ID":"8654c17b-d90b-4b18-929b-cde4eae70e15","Type":"ContainerStarted","Data":"cd974abff70146f4bc02a202aba8b843968df84a68958bbca955f994b04ed522"}
Jul 30 17:47:04.126756 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:47:04.126535    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kv6ls"
Jul 30 17:47:04.152110 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:47:04.152062    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kv6ls" podStartSLOduration=2.632847537 podStartE2EDuration="18.152045095s" totalImagesPullingTime="11.24130223s" totalInitContainerRuntime="4.277895328s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:46:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:47:04.149791307 +0000 UTC m=+2798.637158711" watchObservedRunningTime="2026-07-30 17:47:04.152045095 +0000 UTC m=+2798.639412492"
Jul 30 17:47:05.131530 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:47:05.131499    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kv6ls"
Jul 30 17:47:05.132873 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:47:05.132844    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kv6ls" podUID="8654c17b-d90b-4b18-929b-cde4eae70e15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Jul 30 17:47:06.136283 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:47:06.136243    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kv6ls" podUID="8654c17b-d90b-4b18-929b-cde4eae70e15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Jul 30 17:47:11.140338 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:47:11.140307    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kv6ls"
Jul 30 17:47:11.140817 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:47:11.140790    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kv6ls" podUID="8654c17b-d90b-4b18-929b-cde4eae70e15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Jul 30 17:47:21.141440 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:47:21.141393    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kv6ls" podUID="8654c17b-d90b-4b18-929b-cde4eae70e15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Jul 30 17:47:31.141157 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:47:31.141116    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kv6ls" podUID="8654c17b-d90b-4b18-929b-cde4eae70e15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Jul 30 17:47:35.135779 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:47:35.135743    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:47:41.141118 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:47:41.141077    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kv6ls" podUID="8654c17b-d90b-4b18-929b-cde4eae70e15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Jul 30 17:47:51.141970 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:47:51.141936    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kv6ls"
Jul 30 17:47:58.074738 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:47:58.074705    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kv6ls"]
Jul 30 17:47:58.075171 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:47:58.074927    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kv6ls" podUID="8654c17b-d90b-4b18-929b-cde4eae70e15" containerName="kserve-container" containerID="cri-o://cd974abff70146f4bc02a202aba8b843968df84a68958bbca955f994b04ed522" gracePeriod=30
Jul 30 17:47:58.075171 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:47:58.074947    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kv6ls" podUID="8654c17b-d90b-4b18-929b-cde4eae70e15" containerName="kube-rbac-proxy" containerID="cri-o://87062daf21f9abe6df62504a56488a2b81689e4e7b3f7af8c96c38bd5e724693" gracePeriod=30
Jul 30 17:47:58.087386 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:47:58.087359    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:47:58.386727 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:47:58.386643    2555 generic.go:356] "Generic (PLEG): container finished" podID="8654c17b-d90b-4b18-929b-cde4eae70e15" containerID="87062daf21f9abe6df62504a56488a2b81689e4e7b3f7af8c96c38bd5e724693" exitCode=2
Jul 30 17:47:58.386727 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:47:58.386715    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kv6ls" event={"ID":"8654c17b-d90b-4b18-929b-cde4eae70e15","Type":"ContainerDied","Data":"87062daf21f9abe6df62504a56488a2b81689e4e7b3f7af8c96c38bd5e724693"}
Jul 30 17:48:00.824600 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:48:00.824579    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kv6ls"
Jul 30 17:48:00.941280 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:48:00.941193    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8654c17b-d90b-4b18-929b-cde4eae70e15-kube-rbac-proxy-sar-config\") pod \"8654c17b-d90b-4b18-929b-cde4eae70e15\" (UID: \"8654c17b-d90b-4b18-929b-cde4eae70e15\") "
Jul 30 17:48:00.941280 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:48:00.941236    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8654c17b-d90b-4b18-929b-cde4eae70e15-kserve-provision-location\") pod \"8654c17b-d90b-4b18-929b-cde4eae70e15\" (UID: \"8654c17b-d90b-4b18-929b-cde4eae70e15\") "
Jul 30 17:48:00.941520 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:48:00.941287    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8654c17b-d90b-4b18-929b-cde4eae70e15-proxy-tls\") pod \"8654c17b-d90b-4b18-929b-cde4eae70e15\" (UID: \"8654c17b-d90b-4b18-929b-cde4eae70e15\") "
Jul 30 17:48:00.941520 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:48:00.941337    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jdhmd\" (UniqueName: \"kubernetes.io/projected/8654c17b-d90b-4b18-929b-cde4eae70e15-kube-api-access-jdhmd\") pod \"8654c17b-d90b-4b18-929b-cde4eae70e15\" (UID: \"8654c17b-d90b-4b18-929b-cde4eae70e15\") "
Jul 30 17:48:00.941520 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:48:00.941397    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8654c17b-d90b-4b18-929b-cde4eae70e15-cabundle-cert\") pod \"8654c17b-d90b-4b18-929b-cde4eae70e15\" (UID: \"8654c17b-d90b-4b18-929b-cde4eae70e15\") "
Jul 30 17:48:00.941680 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:48:00.941600    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8654c17b-d90b-4b18-929b-cde4eae70e15-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8654c17b-d90b-4b18-929b-cde4eae70e15" (UID: "8654c17b-d90b-4b18-929b-cde4eae70e15"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:48:00.941786 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:48:00.941761    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8654c17b-d90b-4b18-929b-cde4eae70e15-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8654c17b-d90b-4b18-929b-cde4eae70e15" (UID: "8654c17b-d90b-4b18-929b-cde4eae70e15"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:48:00.943437 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:48:00.943407    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8654c17b-d90b-4b18-929b-cde4eae70e15-kube-api-access-jdhmd" (OuterVolumeSpecName: "kube-api-access-jdhmd") pod "8654c17b-d90b-4b18-929b-cde4eae70e15" (UID: "8654c17b-d90b-4b18-929b-cde4eae70e15"). InnerVolumeSpecName "kube-api-access-jdhmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 17:48:00.943549 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:48:00.943481    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8654c17b-d90b-4b18-929b-cde4eae70e15-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8654c17b-d90b-4b18-929b-cde4eae70e15" (UID: "8654c17b-d90b-4b18-929b-cde4eae70e15"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:48:00.955067 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:48:00.955040    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8654c17b-d90b-4b18-929b-cde4eae70e15-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8654c17b-d90b-4b18-929b-cde4eae70e15" (UID: "8654c17b-d90b-4b18-929b-cde4eae70e15"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 17:48:01.042973 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:48:01.042941    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8654c17b-d90b-4b18-929b-cde4eae70e15-proxy-tls\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:48:01.042973 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:48:01.042968    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdhmd\" (UniqueName: \"kubernetes.io/projected/8654c17b-d90b-4b18-929b-cde4eae70e15-kube-api-access-jdhmd\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:48:01.042973 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:48:01.042979    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8654c17b-d90b-4b18-929b-cde4eae70e15-cabundle-cert\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:48:01.043199 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:48:01.042988    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8654c17b-d90b-4b18-929b-cde4eae70e15-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:48:01.043199 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:48:01.042997    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8654c17b-d90b-4b18-929b-cde4eae70e15-kserve-provision-location\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 17:48:01.401731 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:48:01.401698    2555 generic.go:356] "Generic (PLEG): container finished" podID="8654c17b-d90b-4b18-929b-cde4eae70e15" containerID="cd974abff70146f4bc02a202aba8b843968df84a68958bbca955f994b04ed522" exitCode=0
Jul 30 17:48:01.401911 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:48:01.401753    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kv6ls"
Jul 30 17:48:01.401911 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:48:01.401772    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kv6ls" event={"ID":"8654c17b-d90b-4b18-929b-cde4eae70e15","Type":"ContainerDied","Data":"cd974abff70146f4bc02a202aba8b843968df84a68958bbca955f994b04ed522"}
Jul 30 17:48:01.401911 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:48:01.401834    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kv6ls" event={"ID":"8654c17b-d90b-4b18-929b-cde4eae70e15","Type":"ContainerDied","Data":"ef39f1c9245054269bb409930ffdd84b3a20fc425f2b7aacfdb7b999a5253d93"}
Jul 30 17:48:01.401911 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:48:01.401862    2555 scope.go:117] "RemoveContainer" containerID="87062daf21f9abe6df62504a56488a2b81689e4e7b3f7af8c96c38bd5e724693"
Jul 30 17:48:01.412725 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:48:01.412705    2555 scope.go:117] "RemoveContainer" containerID="cd974abff70146f4bc02a202aba8b843968df84a68958bbca955f994b04ed522"
Jul 30 17:48:01.420510 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:48:01.420487    2555 scope.go:117] "RemoveContainer" containerID="c43fe11ac04dc935f659ccfb23a42cfd0c15dc1fc32e43236522d7b20797d4ae"
Jul 30 17:48:01.428033 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:48:01.427990    2555 scope.go:117] "RemoveContainer" containerID="87062daf21f9abe6df62504a56488a2b81689e4e7b3f7af8c96c38bd5e724693"
Jul 30 17:48:01.428357 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:48:01.428331    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87062daf21f9abe6df62504a56488a2b81689e4e7b3f7af8c96c38bd5e724693\": container with ID starting with 87062daf21f9abe6df62504a56488a2b81689e4e7b3f7af8c96c38bd5e724693 not found: ID does not exist" containerID="87062daf21f9abe6df62504a56488a2b81689e4e7b3f7af8c96c38bd5e724693"
Jul 30 17:48:01.428438 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:48:01.428357    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87062daf21f9abe6df62504a56488a2b81689e4e7b3f7af8c96c38bd5e724693"} err="failed to get container status \"87062daf21f9abe6df62504a56488a2b81689e4e7b3f7af8c96c38bd5e724693\": rpc error: code = NotFound desc = could not find container \"87062daf21f9abe6df62504a56488a2b81689e4e7b3f7af8c96c38bd5e724693\": container with ID starting with 87062daf21f9abe6df62504a56488a2b81689e4e7b3f7af8c96c38bd5e724693 not found: ID does not exist"
Jul 30 17:48:01.428438 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:48:01.428378    2555 scope.go:117] "RemoveContainer" containerID="cd974abff70146f4bc02a202aba8b843968df84a68958bbca955f994b04ed522"
Jul 30 17:48:01.428631 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:48:01.428616    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd974abff70146f4bc02a202aba8b843968df84a68958bbca955f994b04ed522\": container with ID starting with cd974abff70146f4bc02a202aba8b843968df84a68958bbca955f994b04ed522 not found: ID does not exist" containerID="cd974abff70146f4bc02a202aba8b843968df84a68958bbca955f994b04ed522"
Jul 30 17:48:01.428679 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:48:01.428635    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd974abff70146f4bc02a202aba8b843968df84a68958bbca955f994b04ed522"} err="failed to get container status \"cd974abff70146f4bc02a202aba8b843968df84a68958bbca955f994b04ed522\": rpc error: code = NotFound desc = could not find container \"cd974abff70146f4bc02a202aba8b843968df84a68958bbca955f994b04ed522\": container with ID starting with cd974abff70146f4bc02a202aba8b843968df84a68958bbca955f994b04ed522 not found: ID does not exist"
Jul 30 17:48:01.428679 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:48:01.428647    2555 scope.go:117] "RemoveContainer" containerID="c43fe11ac04dc935f659ccfb23a42cfd0c15dc1fc32e43236522d7b20797d4ae"
Jul 30 17:48:01.428850 ip-10-0-143-190 kubenswrapper[2555]: E0730 17:48:01.428832    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c43fe11ac04dc935f659ccfb23a42cfd0c15dc1fc32e43236522d7b20797d4ae\": container with ID starting with c43fe11ac04dc935f659ccfb23a42cfd0c15dc1fc32e43236522d7b20797d4ae not found: ID does not exist" containerID="c43fe11ac04dc935f659ccfb23a42cfd0c15dc1fc32e43236522d7b20797d4ae"
Jul 30 17:48:01.428890 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:48:01.428854    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c43fe11ac04dc935f659ccfb23a42cfd0c15dc1fc32e43236522d7b20797d4ae"} err="failed to get container status \"c43fe11ac04dc935f659ccfb23a42cfd0c15dc1fc32e43236522d7b20797d4ae\": rpc error: code = NotFound desc = could not find container \"c43fe11ac04dc935f659ccfb23a42cfd0c15dc1fc32e43236522d7b20797d4ae\": container with ID starting with c43fe11ac04dc935f659ccfb23a42cfd0c15dc1fc32e43236522d7b20797d4ae not found: ID does not exist"
Jul 30 17:48:01.429985 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:48:01.429967    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kv6ls"]
Jul 30 17:48:01.437306 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:48:01.437282    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kv6ls"]
Jul 30 17:48:02.060440 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:48:02.060407    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8654c17b-d90b-4b18-929b-cde4eae70e15" path="/var/lib/kubelet/pods/8654c17b-d90b-4b18-929b-cde4eae70e15/volumes"
Jul 30 17:48:03.200851 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:48:03.200812    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:48:03.508873 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:48:03.508783    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:49:05.197795 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:49:05.197761    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:49:10.314269 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:49:10.314228    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:49:10.767474 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:49:10.767379    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:50:51.452718 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:50:51.452674    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:50:56.501192 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:50:56.501146    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:50:56.655914 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:50:56.655862    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:52:27.557601 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:52:27.557562    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:52:32.608074 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:52:32.608032    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:52:32.795626 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:52:32.795594    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:53:08.484386 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:53:08.484347    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:54:13.706201 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:54:13.706165    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:54:18.831053 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:54:18.830946    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:54:19.102083 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:54:19.101995    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:55:34.479623 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:55:34.479581    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:56:04.023731 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:56:04.023629    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:56:04.545995 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:56:04.545956    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:56:22.110488 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:56:22.110456    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:56:22.203309 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:56:22.203278    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:56:27.297355 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:56:27.297317    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:56:27.644080 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:56:27.643984    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:56:32.406059 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:56:32.406033    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:56:37.524243 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:56:37.524210    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:56:37.808367 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:56:37.808311    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:56:46.600242 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:56:46.600197    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:58:27.751714 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:58:27.751681    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:58:32.805157 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:58:32.805131    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 17:58:32.990336 ip-10-0-143-190 kubenswrapper[2555]: I0730 17:58:32.990306    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:00:03.159912 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:00:03.159873    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:00:08.139591 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:00:08.139555    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:00:08.281941 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:00:08.281902    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:01:00.843887 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:01:00.843854    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:01:38.330182 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:01:38.330145    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:01:43.442778 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:01:43.442749    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:01:43.619631 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:01:43.619599    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:03:04.395097 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:03:04.395062    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:03:09.508953 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:03:09.508915    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:03:09.651879 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:03:09.651847    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:03:54.998851 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:03:54.998818    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:04:39.751292 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:04:39.751261    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:04:44.796277 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:04:44.796241    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:04:45.403740 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:04:45.403701    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:05:22.069155 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:05:22.069081    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:06:14.980701 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:06:14.980671    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:06:20.075022 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:06:20.074935    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:06:20.219218 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:06:20.219180    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:06:22.179910 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:06:22.179875    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:06:27.291319 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:06:27.291285    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:06:27.389112 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:06:27.389083    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:07:57.474462 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:07:57.474377    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:08:02.524390 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:08:02.524336    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:08:02.585595 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:08:02.585542    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-t22nl"]
Jul 30 18:08:02.585960 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:08:02.585947    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8654c17b-d90b-4b18-929b-cde4eae70e15" containerName="kube-rbac-proxy"
Jul 30 18:08:02.586029 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:08:02.585962    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="8654c17b-d90b-4b18-929b-cde4eae70e15" containerName="kube-rbac-proxy"
Jul 30 18:08:02.586029 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:08:02.585986    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8654c17b-d90b-4b18-929b-cde4eae70e15" containerName="kserve-container"
Jul 30 18:08:02.586029 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:08:02.585992    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="8654c17b-d90b-4b18-929b-cde4eae70e15" containerName="kserve-container"
Jul 30 18:08:02.586029 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:08:02.586000    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8654c17b-d90b-4b18-929b-cde4eae70e15" containerName="storage-initializer"
Jul 30 18:08:02.586029 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:08:02.586017    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="8654c17b-d90b-4b18-929b-cde4eae70e15" containerName="storage-initializer"
Jul 30 18:08:02.586196 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:08:02.586075    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="8654c17b-d90b-4b18-929b-cde4eae70e15" containerName="kube-rbac-proxy"
Jul 30 18:08:02.586196 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:08:02.586083    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="8654c17b-d90b-4b18-929b-cde4eae70e15" containerName="kserve-container"
Jul 30 18:08:02.589269 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:08:02.589253    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-t22nl"
Jul 30 18:08:02.591997 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:08:02.591979    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 30 18:08:02.592138 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:08:02.591997    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 30 18:08:02.592138 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:08:02.592002    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\""
Jul 30 18:08:02.592138 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:08:02.592115    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-plw5t\""
Jul 30 18:08:02.592312 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:08:02.592146    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-predictor-serving-cert\""
Jul 30 18:08:02.592352 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:08:02.592335    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 30 18:08:02.593168 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:08:02.593151    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 30 18:08:02.600139 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:08:02.600091    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-t22nl"]
Jul 30 18:08:02.666897 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:08:02.666863    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc811ecd-5b69-4301-ab3b-f8a4d321a8a2-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-t22nl\" (UID: \"dc811ecd-5b69-4301-ab3b-f8a4d321a8a2\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-t22nl"
Jul 30 18:08:02.667108 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:08:02.666909    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc811ecd-5b69-4301-ab3b-f8a4d321a8a2-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-t22nl\" (UID: \"dc811ecd-5b69-4301-ab3b-f8a4d321a8a2\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-t22nl"
Jul 30 18:08:02.667108 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:08:02.667038    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc811ecd-5b69-4301-ab3b-f8a4d321a8a2-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-t22nl\" (UID: \"dc811ecd-5b69-4301-ab3b-f8a4d321a8a2\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-t22nl"
Jul 30 18:08:02.667108 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:08:02.667066    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lckt4\" (UniqueName: \"kubernetes.io/projected/dc811ecd-5b69-4301-ab3b-f8a4d321a8a2-kube-api-access-lckt4\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-t22nl\" (UID: \"dc811ecd-5b69-4301-ab3b-f8a4d321a8a2\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-t22nl"
Jul 30 18:08:02.667225 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:08:02.667116    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc811ecd-5b69-4301-ab3b-f8a4d321a8a2-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-t22nl\" (UID: \"dc811ecd-5b69-4301-ab3b-f8a4d321a8a2\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-t22nl"
Jul 30 18:08:02.768114 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:08:02.768076    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc811ecd-5b69-4301-ab3b-f8a4d321a8a2-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-t22nl\" (UID: \"dc811ecd-5b69-4301-ab3b-f8a4d321a8a2\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-t22nl"
Jul 30 18:08:02.768114 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:08:02.768122    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lckt4\" (UniqueName: \"kubernetes.io/projected/dc811ecd-5b69-4301-ab3b-f8a4d321a8a2-kube-api-access-lckt4\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-t22nl\" (UID: \"dc811ecd-5b69-4301-ab3b-f8a4d321a8a2\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-t22nl"
Jul 30 18:08:02.768401 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:08:02.768250    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc811ecd-5b69-4301-ab3b-f8a4d321a8a2-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-t22nl\" (UID: \"dc811ecd-5b69-4301-ab3b-f8a4d321a8a2\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-t22nl"
Jul 30 18:08:02.768401 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:08:02.768306    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc811ecd-5b69-4301-ab3b-f8a4d321a8a2-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-t22nl\" (UID: \"dc811ecd-5b69-4301-ab3b-f8a4d321a8a2\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-t22nl"
Jul 30 18:08:02.768401 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:08:02.768345    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc811ecd-5b69-4301-ab3b-f8a4d321a8a2-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-t22nl\" (UID: \"dc811ecd-5b69-4301-ab3b-f8a4d321a8a2\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-t22nl"
Jul 30 18:08:02.768538 ip-10-0-143-190 kubenswrapper[2555]: E0730 18:08:02.768445    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-serving-cert: secret "sklearn-v2-mlserver-predictor-serving-cert" not found
Jul 30 18:08:02.768538 ip-10-0-143-190 kubenswrapper[2555]: E0730 18:08:02.768532    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc811ecd-5b69-4301-ab3b-f8a4d321a8a2-proxy-tls podName:dc811ecd-5b69-4301-ab3b-f8a4d321a8a2 nodeName:}" failed. No retries permitted until 2026-07-30 18:08:03.268509474 +0000 UTC m=+4057.755876854 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/dc811ecd-5b69-4301-ab3b-f8a4d321a8a2-proxy-tls") pod "sklearn-v2-mlserver-predictor-7f568cdc6b-t22nl" (UID: "dc811ecd-5b69-4301-ab3b-f8a4d321a8a2") : secret "sklearn-v2-mlserver-predictor-serving-cert" not found
Jul 30 18:08:02.768744 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:08:02.768724    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc811ecd-5b69-4301-ab3b-f8a4d321a8a2-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-t22nl\" (UID: \"dc811ecd-5b69-4301-ab3b-f8a4d321a8a2\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-t22nl"
Jul 30 18:08:02.768802 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:08:02.768787    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc811ecd-5b69-4301-ab3b-f8a4d321a8a2-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-t22nl\" (UID: \"dc811ecd-5b69-4301-ab3b-f8a4d321a8a2\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-t22nl"
Jul 30 18:08:02.768839 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:08:02.768815    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc811ecd-5b69-4301-ab3b-f8a4d321a8a2-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-t22nl\" (UID: \"dc811ecd-5b69-4301-ab3b-f8a4d321a8a2\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-t22nl"
Jul 30 18:08:02.780837 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:08:02.780786    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lckt4\" (UniqueName: \"kubernetes.io/projected/dc811ecd-5b69-4301-ab3b-f8a4d321a8a2-kube-api-access-lckt4\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-t22nl\" (UID: \"dc811ecd-5b69-4301-ab3b-f8a4d321a8a2\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-t22nl"
Jul 30 18:08:02.955584 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:08:02.955552    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:08:03.271899 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:08:03.271868    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc811ecd-5b69-4301-ab3b-f8a4d321a8a2-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-t22nl\" (UID: \"dc811ecd-5b69-4301-ab3b-f8a4d321a8a2\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-t22nl"
Jul 30 18:08:03.274342 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:08:03.274315    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc811ecd-5b69-4301-ab3b-f8a4d321a8a2-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-t22nl\" (UID: \"dc811ecd-5b69-4301-ab3b-f8a4d321a8a2\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-t22nl"
Jul 30 18:08:03.500874 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:08:03.500843    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-t22nl"
Jul 30 18:08:03.633657 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:08:03.633631    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-t22nl"]
Jul 30 18:08:03.637760 ip-10-0-143-190 kubenswrapper[2555]: W0730 18:08:03.637728    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc811ecd_5b69_4301_ab3b_f8a4d321a8a2.slice/crio-93e05aa1fdb0fc08aaac73b6aa7dd8d85881ca262d054cbf816a4e2aba9ebbf1 WatchSource:0}: Error finding container 93e05aa1fdb0fc08aaac73b6aa7dd8d85881ca262d054cbf816a4e2aba9ebbf1: Status 404 returned error can't find the container with id 93e05aa1fdb0fc08aaac73b6aa7dd8d85881ca262d054cbf816a4e2aba9ebbf1
Jul 30 18:08:03.834187 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:08:03.834106    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-t22nl" event={"ID":"dc811ecd-5b69-4301-ab3b-f8a4d321a8a2","Type":"ContainerStarted","Data":"37cf25f8b4db48347bf03248d57cb9de23ab18cb476411ec53be073e84a64d5c"}
Jul 30 18:08:03.834187 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:08:03.834170    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-t22nl" event={"ID":"dc811ecd-5b69-4301-ab3b-f8a4d321a8a2","Type":"ContainerStarted","Data":"93e05aa1fdb0fc08aaac73b6aa7dd8d85881ca262d054cbf816a4e2aba9ebbf1"}
Jul 30 18:08:07.852329 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:08:07.852299    2555 generic.go:356] "Generic (PLEG): container finished" podID="dc811ecd-5b69-4301-ab3b-f8a4d321a8a2" containerID="37cf25f8b4db48347bf03248d57cb9de23ab18cb476411ec53be073e84a64d5c" exitCode=0
Jul 30 18:08:07.852737 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:08:07.852336    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-t22nl" event={"ID":"dc811ecd-5b69-4301-ab3b-f8a4d321a8a2","Type":"ContainerDied","Data":"37cf25f8b4db48347bf03248d57cb9de23ab18cb476411ec53be073e84a64d5c"}
Jul 30 18:08:08.858779 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:08:08.858742    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-t22nl" event={"ID":"dc811ecd-5b69-4301-ab3b-f8a4d321a8a2","Type":"ContainerStarted","Data":"6fc5ccc314df248cdd19dc0acc1b2d5788501d68259e6839a9b37a0e8a7c3f71"}
Jul 30 18:08:08.859186 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:08:08.858803    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-t22nl" event={"ID":"dc811ecd-5b69-4301-ab3b-f8a4d321a8a2","Type":"ContainerStarted","Data":"9124fd03d85ba4bef2149f3a53c16eddd2ab247e6d52b1c009c676e0ab4dae62"}
Jul 30 18:08:08.859186 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:08:08.859124    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-t22nl"
Jul 30 18:08:08.859261 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:08:08.859195    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-t22nl"
Jul 30 18:08:08.880690 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:08:08.880639    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-t22nl" podStartSLOduration=3.196029757 podStartE2EDuration="6.880604723s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.684574966s" isStatefulPod=false podCreationTimestamp="2026-07-30 18:08:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 18:08:08.880185358 +0000 UTC m=+4063.367552804" watchObservedRunningTime="2026-07-30 18:08:08.880604723 +0000 UTC m=+4063.367972126"
Jul 30 18:08:14.870774 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:08:14.870747    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-t22nl"
Jul 30 18:08:44.903156 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:08:44.903110    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-t22nl" podUID="dc811ecd-5b69-4301-ab3b-f8a4d321a8a2" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Jul 30 18:08:54.873896 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:08:54.873864    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-t22nl"
Jul 30 18:09:00.339553 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:00.339520    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:09:02.690674 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:02.690644    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-t22nl"]
Jul 30 18:09:02.691098 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:02.690947    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-t22nl" podUID="dc811ecd-5b69-4301-ab3b-f8a4d321a8a2" containerName="kube-rbac-proxy" containerID="cri-o://6fc5ccc314df248cdd19dc0acc1b2d5788501d68259e6839a9b37a0e8a7c3f71" gracePeriod=30
Jul 30 18:09:02.691098 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:02.690932    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-t22nl" podUID="dc811ecd-5b69-4301-ab3b-f8a4d321a8a2" containerName="kserve-container" containerID="cri-o://9124fd03d85ba4bef2149f3a53c16eddd2ab247e6d52b1c009c676e0ab4dae62" gracePeriod=30
Jul 30 18:09:02.718329 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:02.718289    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:09:03.115568 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:03.115527    2555 generic.go:356] "Generic (PLEG): container finished" podID="dc811ecd-5b69-4301-ab3b-f8a4d321a8a2" containerID="6fc5ccc314df248cdd19dc0acc1b2d5788501d68259e6839a9b37a0e8a7c3f71" exitCode=2
Jul 30 18:09:03.115751 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:03.115587    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-t22nl" event={"ID":"dc811ecd-5b69-4301-ab3b-f8a4d321a8a2","Type":"ContainerDied","Data":"6fc5ccc314df248cdd19dc0acc1b2d5788501d68259e6839a9b37a0e8a7c3f71"}
Jul 30 18:09:04.866223 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:04.866182    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-t22nl" podUID="dc811ecd-5b69-4301-ab3b-f8a4d321a8a2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.48:8643/healthz\": dial tcp 10.133.0.48:8643: connect: connection refused"
Jul 30 18:09:05.913238 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:05.913155    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-t22nl" podUID="dc811ecd-5b69-4301-ab3b-f8a4d321a8a2" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.48:8080/v2/models/sklearn-v2-mlserver/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 30 18:09:07.814935 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:07.814900    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:09:08.150304 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:08.150215    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:09:09.942002 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:09.941979    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-t22nl"
Jul 30 18:09:10.055928 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:10.055897    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc811ecd-5b69-4301-ab3b-f8a4d321a8a2-proxy-tls\") pod \"dc811ecd-5b69-4301-ab3b-f8a4d321a8a2\" (UID: \"dc811ecd-5b69-4301-ab3b-f8a4d321a8a2\") "
Jul 30 18:09:10.056135 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:10.055968    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc811ecd-5b69-4301-ab3b-f8a4d321a8a2-kube-rbac-proxy-sar-config\") pod \"dc811ecd-5b69-4301-ab3b-f8a4d321a8a2\" (UID: \"dc811ecd-5b69-4301-ab3b-f8a4d321a8a2\") "
Jul 30 18:09:10.056135 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:10.055992    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc811ecd-5b69-4301-ab3b-f8a4d321a8a2-kserve-provision-location\") pod \"dc811ecd-5b69-4301-ab3b-f8a4d321a8a2\" (UID: \"dc811ecd-5b69-4301-ab3b-f8a4d321a8a2\") "
Jul 30 18:09:10.056135 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:10.056099    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc811ecd-5b69-4301-ab3b-f8a4d321a8a2-cabundle-cert\") pod \"dc811ecd-5b69-4301-ab3b-f8a4d321a8a2\" (UID: \"dc811ecd-5b69-4301-ab3b-f8a4d321a8a2\") "
Jul 30 18:09:10.056312 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:10.056165    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lckt4\" (UniqueName: \"kubernetes.io/projected/dc811ecd-5b69-4301-ab3b-f8a4d321a8a2-kube-api-access-lckt4\") pod \"dc811ecd-5b69-4301-ab3b-f8a4d321a8a2\" (UID: \"dc811ecd-5b69-4301-ab3b-f8a4d321a8a2\") "
Jul 30 18:09:10.056370 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:10.056315    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc811ecd-5b69-4301-ab3b-f8a4d321a8a2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dc811ecd-5b69-4301-ab3b-f8a4d321a8a2" (UID: "dc811ecd-5b69-4301-ab3b-f8a4d321a8a2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 18:09:10.056419 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:10.056381    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc811ecd-5b69-4301-ab3b-f8a4d321a8a2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "dc811ecd-5b69-4301-ab3b-f8a4d321a8a2" (UID: "dc811ecd-5b69-4301-ab3b-f8a4d321a8a2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 18:09:10.056457 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:10.056445    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc811ecd-5b69-4301-ab3b-f8a4d321a8a2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 18:09:10.056506 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:10.056457    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc811ecd-5b69-4301-ab3b-f8a4d321a8a2-kserve-provision-location\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 18:09:10.056706 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:10.056681    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc811ecd-5b69-4301-ab3b-f8a4d321a8a2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "dc811ecd-5b69-4301-ab3b-f8a4d321a8a2" (UID: "dc811ecd-5b69-4301-ab3b-f8a4d321a8a2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 18:09:10.058141 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:10.058119    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc811ecd-5b69-4301-ab3b-f8a4d321a8a2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dc811ecd-5b69-4301-ab3b-f8a4d321a8a2" (UID: "dc811ecd-5b69-4301-ab3b-f8a4d321a8a2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 18:09:10.058569 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:10.058548    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc811ecd-5b69-4301-ab3b-f8a4d321a8a2-kube-api-access-lckt4" (OuterVolumeSpecName: "kube-api-access-lckt4") pod "dc811ecd-5b69-4301-ab3b-f8a4d321a8a2" (UID: "dc811ecd-5b69-4301-ab3b-f8a4d321a8a2"). InnerVolumeSpecName "kube-api-access-lckt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 18:09:10.147041 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:10.146989    2555 generic.go:356] "Generic (PLEG): container finished" podID="dc811ecd-5b69-4301-ab3b-f8a4d321a8a2" containerID="9124fd03d85ba4bef2149f3a53c16eddd2ab247e6d52b1c009c676e0ab4dae62" exitCode=0
Jul 30 18:09:10.147216 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:10.147067    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-t22nl"
Jul 30 18:09:10.147216 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:10.147079    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-t22nl" event={"ID":"dc811ecd-5b69-4301-ab3b-f8a4d321a8a2","Type":"ContainerDied","Data":"9124fd03d85ba4bef2149f3a53c16eddd2ab247e6d52b1c009c676e0ab4dae62"}
Jul 30 18:09:10.147216 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:10.147146    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-t22nl" event={"ID":"dc811ecd-5b69-4301-ab3b-f8a4d321a8a2","Type":"ContainerDied","Data":"93e05aa1fdb0fc08aaac73b6aa7dd8d85881ca262d054cbf816a4e2aba9ebbf1"}
Jul 30 18:09:10.147216 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:10.147172    2555 scope.go:117] "RemoveContainer" containerID="6fc5ccc314df248cdd19dc0acc1b2d5788501d68259e6839a9b37a0e8a7c3f71"
Jul 30 18:09:10.155750 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:10.155732    2555 scope.go:117] "RemoveContainer" containerID="9124fd03d85ba4bef2149f3a53c16eddd2ab247e6d52b1c009c676e0ab4dae62"
Jul 30 18:09:10.156878 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:10.156860    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc811ecd-5b69-4301-ab3b-f8a4d321a8a2-proxy-tls\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 18:09:10.156955 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:10.156885    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc811ecd-5b69-4301-ab3b-f8a4d321a8a2-cabundle-cert\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 18:09:10.156955 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:10.156899    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lckt4\" (UniqueName: \"kubernetes.io/projected/dc811ecd-5b69-4301-ab3b-f8a4d321a8a2-kube-api-access-lckt4\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 18:09:10.163539 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:10.163519    2555 scope.go:117] "RemoveContainer" containerID="37cf25f8b4db48347bf03248d57cb9de23ab18cb476411ec53be073e84a64d5c"
Jul 30 18:09:10.170151 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:10.170127    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-t22nl"]
Jul 30 18:09:10.172358 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:10.172335    2555 scope.go:117] "RemoveContainer" containerID="6fc5ccc314df248cdd19dc0acc1b2d5788501d68259e6839a9b37a0e8a7c3f71"
Jul 30 18:09:10.172622 ip-10-0-143-190 kubenswrapper[2555]: E0730 18:09:10.172604    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fc5ccc314df248cdd19dc0acc1b2d5788501d68259e6839a9b37a0e8a7c3f71\": container with ID starting with 6fc5ccc314df248cdd19dc0acc1b2d5788501d68259e6839a9b37a0e8a7c3f71 not found: ID does not exist" containerID="6fc5ccc314df248cdd19dc0acc1b2d5788501d68259e6839a9b37a0e8a7c3f71"
Jul 30 18:09:10.172697 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:10.172629    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fc5ccc314df248cdd19dc0acc1b2d5788501d68259e6839a9b37a0e8a7c3f71"} err="failed to get container status \"6fc5ccc314df248cdd19dc0acc1b2d5788501d68259e6839a9b37a0e8a7c3f71\": rpc error: code = NotFound desc = could not find container \"6fc5ccc314df248cdd19dc0acc1b2d5788501d68259e6839a9b37a0e8a7c3f71\": container with ID starting with 6fc5ccc314df248cdd19dc0acc1b2d5788501d68259e6839a9b37a0e8a7c3f71 not found: ID does not exist"
Jul 30 18:09:10.172697 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:10.172648    2555 scope.go:117] "RemoveContainer" containerID="9124fd03d85ba4bef2149f3a53c16eddd2ab247e6d52b1c009c676e0ab4dae62"
Jul 30 18:09:10.172912 ip-10-0-143-190 kubenswrapper[2555]: E0730 18:09:10.172896    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9124fd03d85ba4bef2149f3a53c16eddd2ab247e6d52b1c009c676e0ab4dae62\": container with ID starting with 9124fd03d85ba4bef2149f3a53c16eddd2ab247e6d52b1c009c676e0ab4dae62 not found: ID does not exist" containerID="9124fd03d85ba4bef2149f3a53c16eddd2ab247e6d52b1c009c676e0ab4dae62"
Jul 30 18:09:10.172985 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:10.172916    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9124fd03d85ba4bef2149f3a53c16eddd2ab247e6d52b1c009c676e0ab4dae62"} err="failed to get container status \"9124fd03d85ba4bef2149f3a53c16eddd2ab247e6d52b1c009c676e0ab4dae62\": rpc error: code = NotFound desc = could not find container \"9124fd03d85ba4bef2149f3a53c16eddd2ab247e6d52b1c009c676e0ab4dae62\": container with ID starting with 9124fd03d85ba4bef2149f3a53c16eddd2ab247e6d52b1c009c676e0ab4dae62 not found: ID does not exist"
Jul 30 18:09:10.172985 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:10.172930    2555 scope.go:117] "RemoveContainer" containerID="37cf25f8b4db48347bf03248d57cb9de23ab18cb476411ec53be073e84a64d5c"
Jul 30 18:09:10.173165 ip-10-0-143-190 kubenswrapper[2555]: E0730 18:09:10.173148    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37cf25f8b4db48347bf03248d57cb9de23ab18cb476411ec53be073e84a64d5c\": container with ID starting with 37cf25f8b4db48347bf03248d57cb9de23ab18cb476411ec53be073e84a64d5c not found: ID does not exist" containerID="37cf25f8b4db48347bf03248d57cb9de23ab18cb476411ec53be073e84a64d5c"
Jul 30 18:09:10.173224 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:10.173179    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37cf25f8b4db48347bf03248d57cb9de23ab18cb476411ec53be073e84a64d5c"} err="failed to get container status \"37cf25f8b4db48347bf03248d57cb9de23ab18cb476411ec53be073e84a64d5c\": rpc error: code = NotFound desc = could not find container \"37cf25f8b4db48347bf03248d57cb9de23ab18cb476411ec53be073e84a64d5c\": container with ID starting with 37cf25f8b4db48347bf03248d57cb9de23ab18cb476411ec53be073e84a64d5c not found: ID does not exist"
Jul 30 18:09:10.173878 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:10.173856    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-t22nl"]
Jul 30 18:09:10.866096 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:10.866049    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-t22nl" podUID="dc811ecd-5b69-4301-ab3b-f8a4d321a8a2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.48:8643/healthz\": context deadline exceeded"
Jul 30 18:09:12.060997 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:12.060965    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc811ecd-5b69-4301-ab3b-f8a4d321a8a2" path="/var/lib/kubelet/pods/dc811ecd-5b69-4301-ab3b-f8a4d321a8a2/volumes"
Jul 30 18:09:39.791260 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:39.791222    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:09:44.905623 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:44.905584    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-svrxw"]
Jul 30 18:09:44.906023 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:44.905965    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc811ecd-5b69-4301-ab3b-f8a4d321a8a2" containerName="storage-initializer"
Jul 30 18:09:44.906023 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:44.905977    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc811ecd-5b69-4301-ab3b-f8a4d321a8a2" containerName="storage-initializer"
Jul 30 18:09:44.906023 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:44.905986    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc811ecd-5b69-4301-ab3b-f8a4d321a8a2" containerName="kserve-container"
Jul 30 18:09:44.906023 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:44.905994    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc811ecd-5b69-4301-ab3b-f8a4d321a8a2" containerName="kserve-container"
Jul 30 18:09:44.906162 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:44.906036    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc811ecd-5b69-4301-ab3b-f8a4d321a8a2" containerName="kube-rbac-proxy"
Jul 30 18:09:44.906162 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:44.906043    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc811ecd-5b69-4301-ab3b-f8a4d321a8a2" containerName="kube-rbac-proxy"
Jul 30 18:09:44.906162 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:44.906107    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc811ecd-5b69-4301-ab3b-f8a4d321a8a2" containerName="kserve-container"
Jul 30 18:09:44.906162 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:44.906116    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc811ecd-5b69-4301-ab3b-f8a4d321a8a2" containerName="kube-rbac-proxy"
Jul 30 18:09:44.909366 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:44.909337    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-svrxw"
Jul 30 18:09:44.912375 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:44.912353    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\""
Jul 30 18:09:44.912375 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:44.912356    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 30 18:09:44.912375 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:44.912403    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-predictor-serving-cert\""
Jul 30 18:09:44.912707 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:44.912377    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 30 18:09:44.912707 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:44.912377    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 30 18:09:44.913587 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:44.913547    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-plw5t\""
Jul 30 18:09:44.913587 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:44.913577    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 30 18:09:44.921149 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:44.921127    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-svrxw"]
Jul 30 18:09:45.057590 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:45.057542    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0353158b-2e9f-4fb9-bfa7-c5dc31dd586a-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-svrxw\" (UID: \"0353158b-2e9f-4fb9-bfa7-c5dc31dd586a\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-svrxw"
Jul 30 18:09:45.057779 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:45.057603    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzdrd\" (UniqueName: \"kubernetes.io/projected/0353158b-2e9f-4fb9-bfa7-c5dc31dd586a-kube-api-access-jzdrd\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-svrxw\" (UID: \"0353158b-2e9f-4fb9-bfa7-c5dc31dd586a\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-svrxw"
Jul 30 18:09:45.057779 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:45.057702    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0353158b-2e9f-4fb9-bfa7-c5dc31dd586a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-svrxw\" (UID: \"0353158b-2e9f-4fb9-bfa7-c5dc31dd586a\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-svrxw"
Jul 30 18:09:45.057779 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:45.057760    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0353158b-2e9f-4fb9-bfa7-c5dc31dd586a-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-svrxw\" (UID: \"0353158b-2e9f-4fb9-bfa7-c5dc31dd586a\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-svrxw"
Jul 30 18:09:45.057948 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:45.057851    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0353158b-2e9f-4fb9-bfa7-c5dc31dd586a-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-svrxw\" (UID: \"0353158b-2e9f-4fb9-bfa7-c5dc31dd586a\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-svrxw"
Jul 30 18:09:45.081281 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:45.081251    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:09:45.159415 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:45.159316    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0353158b-2e9f-4fb9-bfa7-c5dc31dd586a-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-svrxw\" (UID: \"0353158b-2e9f-4fb9-bfa7-c5dc31dd586a\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-svrxw"
Jul 30 18:09:45.159415 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:45.159381    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzdrd\" (UniqueName: \"kubernetes.io/projected/0353158b-2e9f-4fb9-bfa7-c5dc31dd586a-kube-api-access-jzdrd\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-svrxw\" (UID: \"0353158b-2e9f-4fb9-bfa7-c5dc31dd586a\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-svrxw"
Jul 30 18:09:45.159651 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:45.159427    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0353158b-2e9f-4fb9-bfa7-c5dc31dd586a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-svrxw\" (UID: \"0353158b-2e9f-4fb9-bfa7-c5dc31dd586a\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-svrxw"
Jul 30 18:09:45.159651 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:45.159470    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0353158b-2e9f-4fb9-bfa7-c5dc31dd586a-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-svrxw\" (UID: \"0353158b-2e9f-4fb9-bfa7-c5dc31dd586a\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-svrxw"
Jul 30 18:09:45.159651 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:45.159628    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0353158b-2e9f-4fb9-bfa7-c5dc31dd586a-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-svrxw\" (UID: \"0353158b-2e9f-4fb9-bfa7-c5dc31dd586a\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-svrxw"
Jul 30 18:09:45.160033 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:45.159976    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0353158b-2e9f-4fb9-bfa7-c5dc31dd586a-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-svrxw\" (UID: \"0353158b-2e9f-4fb9-bfa7-c5dc31dd586a\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-svrxw"
Jul 30 18:09:45.160146 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:45.160134    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0353158b-2e9f-4fb9-bfa7-c5dc31dd586a-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-svrxw\" (UID: \"0353158b-2e9f-4fb9-bfa7-c5dc31dd586a\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-svrxw"
Jul 30 18:09:45.160301 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:45.160282    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0353158b-2e9f-4fb9-bfa7-c5dc31dd586a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-svrxw\" (UID: \"0353158b-2e9f-4fb9-bfa7-c5dc31dd586a\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-svrxw"
Jul 30 18:09:45.162337 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:45.162315    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0353158b-2e9f-4fb9-bfa7-c5dc31dd586a-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-svrxw\" (UID: \"0353158b-2e9f-4fb9-bfa7-c5dc31dd586a\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-svrxw"
Jul 30 18:09:45.169173 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:45.166839    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:09:45.173091 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:45.170747    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzdrd\" (UniqueName: \"kubernetes.io/projected/0353158b-2e9f-4fb9-bfa7-c5dc31dd586a-kube-api-access-jzdrd\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-svrxw\" (UID: \"0353158b-2e9f-4fb9-bfa7-c5dc31dd586a\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-svrxw"
Jul 30 18:09:45.220417 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:45.220391    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-svrxw"
Jul 30 18:09:45.352731 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:45.352707    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-svrxw"]
Jul 30 18:09:45.354404 ip-10-0-143-190 kubenswrapper[2555]: W0730 18:09:45.354378    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0353158b_2e9f_4fb9_bfa7_c5dc31dd586a.slice/crio-b082e856e0831d7580c0dc0a38bd477ff98f0325768c87a9e568044000f2bc8d WatchSource:0}: Error finding container b082e856e0831d7580c0dc0a38bd477ff98f0325768c87a9e568044000f2bc8d: Status 404 returned error can't find the container with id b082e856e0831d7580c0dc0a38bd477ff98f0325768c87a9e568044000f2bc8d
Jul 30 18:09:46.310756 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:46.310722    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-svrxw" event={"ID":"0353158b-2e9f-4fb9-bfa7-c5dc31dd586a","Type":"ContainerStarted","Data":"88e39645121230431db938b5886846365896b5e8563cb5f7c8217606f83203e3"}
Jul 30 18:09:46.311202 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:46.310782    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-svrxw" event={"ID":"0353158b-2e9f-4fb9-bfa7-c5dc31dd586a","Type":"ContainerStarted","Data":"b082e856e0831d7580c0dc0a38bd477ff98f0325768c87a9e568044000f2bc8d"}
Jul 30 18:09:49.327245 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:49.327156    2555 generic.go:356] "Generic (PLEG): container finished" podID="0353158b-2e9f-4fb9-bfa7-c5dc31dd586a" containerID="88e39645121230431db938b5886846365896b5e8563cb5f7c8217606f83203e3" exitCode=0
Jul 30 18:09:49.327245 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:49.327231    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-svrxw" event={"ID":"0353158b-2e9f-4fb9-bfa7-c5dc31dd586a","Type":"ContainerDied","Data":"88e39645121230431db938b5886846365896b5e8563cb5f7c8217606f83203e3"}
Jul 30 18:09:50.333905 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:50.333869    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-svrxw" event={"ID":"0353158b-2e9f-4fb9-bfa7-c5dc31dd586a","Type":"ContainerStarted","Data":"86b98969428f56d95a60ae8b43f0e1e7b53f648025b3b77fe83a027defc17372"}
Jul 30 18:09:50.334305 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:50.333936    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-svrxw" event={"ID":"0353158b-2e9f-4fb9-bfa7-c5dc31dd586a","Type":"ContainerStarted","Data":"7086c27cbf3eaa4a85661c4e250e08becc65d5ff64af2d239353e43fa7ee0bdf"}
Jul 30 18:09:50.334305 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:50.334138    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-svrxw"
Jul 30 18:09:50.356333 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:50.356281    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-svrxw" podStartSLOduration=2.7262868879999997 podStartE2EDuration="6.356266107s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.629979219s" isStatefulPod=false podCreationTimestamp="2026-07-30 18:09:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 18:09:50.355544788 +0000 UTC m=+4164.842912191" watchObservedRunningTime="2026-07-30 18:09:50.356266107 +0000 UTC m=+4164.843633508"
Jul 30 18:09:51.339024 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:51.338976    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-svrxw"
Jul 30 18:09:57.348812 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:09:57.348780    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-svrxw"
Jul 30 18:10:27.403430 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:10:27.403380    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-svrxw" podUID="0353158b-2e9f-4fb9-bfa7-c5dc31dd586a" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Jul 30 18:10:37.352081 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:10:37.352046    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-svrxw"
Jul 30 18:10:45.033265 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:10:45.033235    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-svrxw"]
Jul 30 18:10:45.033697 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:10:45.033521    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-svrxw" podUID="0353158b-2e9f-4fb9-bfa7-c5dc31dd586a" containerName="kserve-container" containerID="cri-o://7086c27cbf3eaa4a85661c4e250e08becc65d5ff64af2d239353e43fa7ee0bdf" gracePeriod=30
Jul 30 18:10:45.033697 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:10:45.033564    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-svrxw" podUID="0353158b-2e9f-4fb9-bfa7-c5dc31dd586a" containerName="kube-rbac-proxy" containerID="cri-o://86b98969428f56d95a60ae8b43f0e1e7b53f648025b3b77fe83a027defc17372" gracePeriod=30
Jul 30 18:10:45.054455 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:10:45.054426    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:10:45.593333 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:10:45.593301    2555 generic.go:356] "Generic (PLEG): container finished" podID="0353158b-2e9f-4fb9-bfa7-c5dc31dd586a" containerID="86b98969428f56d95a60ae8b43f0e1e7b53f648025b3b77fe83a027defc17372" exitCode=2
Jul 30 18:10:45.593507 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:10:45.593378    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-svrxw" event={"ID":"0353158b-2e9f-4fb9-bfa7-c5dc31dd586a","Type":"ContainerDied","Data":"86b98969428f56d95a60ae8b43f0e1e7b53f648025b3b77fe83a027defc17372"}
Jul 30 18:10:47.344696 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:10:47.344592    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-svrxw" podUID="0353158b-2e9f-4fb9-bfa7-c5dc31dd586a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.49:8643/healthz\": dial tcp 10.133.0.49:8643: connect: connection refused"
Jul 30 18:10:48.391212 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:10:48.391164    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-svrxw" podUID="0353158b-2e9f-4fb9-bfa7-c5dc31dd586a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.49:8080/v2/models/isvc-sklearn-v2-runtime/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 30 18:10:50.183076 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:10:50.183039    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:10:50.485040 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:10:50.484937    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:10:52.195708 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:10:52.195686    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-svrxw"
Jul 30 18:10:52.249801 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:10:52.249765    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0353158b-2e9f-4fb9-bfa7-c5dc31dd586a-kube-rbac-proxy-sar-config\") pod \"0353158b-2e9f-4fb9-bfa7-c5dc31dd586a\" (UID: \"0353158b-2e9f-4fb9-bfa7-c5dc31dd586a\") "
Jul 30 18:10:52.249993 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:10:52.249848    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jzdrd\" (UniqueName: \"kubernetes.io/projected/0353158b-2e9f-4fb9-bfa7-c5dc31dd586a-kube-api-access-jzdrd\") pod \"0353158b-2e9f-4fb9-bfa7-c5dc31dd586a\" (UID: \"0353158b-2e9f-4fb9-bfa7-c5dc31dd586a\") "
Jul 30 18:10:52.249993 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:10:52.249910    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0353158b-2e9f-4fb9-bfa7-c5dc31dd586a-kserve-provision-location\") pod \"0353158b-2e9f-4fb9-bfa7-c5dc31dd586a\" (UID: \"0353158b-2e9f-4fb9-bfa7-c5dc31dd586a\") "
Jul 30 18:10:52.249993 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:10:52.249966    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0353158b-2e9f-4fb9-bfa7-c5dc31dd586a-cabundle-cert\") pod \"0353158b-2e9f-4fb9-bfa7-c5dc31dd586a\" (UID: \"0353158b-2e9f-4fb9-bfa7-c5dc31dd586a\") "
Jul 30 18:10:52.250198 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:10:52.249999    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0353158b-2e9f-4fb9-bfa7-c5dc31dd586a-proxy-tls\") pod \"0353158b-2e9f-4fb9-bfa7-c5dc31dd586a\" (UID: \"0353158b-2e9f-4fb9-bfa7-c5dc31dd586a\") "
Jul 30 18:10:52.250264 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:10:52.250240    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0353158b-2e9f-4fb9-bfa7-c5dc31dd586a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0353158b-2e9f-4fb9-bfa7-c5dc31dd586a" (UID: "0353158b-2e9f-4fb9-bfa7-c5dc31dd586a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 18:10:52.250315 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:10:52.250293    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0353158b-2e9f-4fb9-bfa7-c5dc31dd586a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0353158b-2e9f-4fb9-bfa7-c5dc31dd586a" (UID: "0353158b-2e9f-4fb9-bfa7-c5dc31dd586a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 18:10:52.250376 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:10:52.250348    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0353158b-2e9f-4fb9-bfa7-c5dc31dd586a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0353158b-2e9f-4fb9-bfa7-c5dc31dd586a" (UID: "0353158b-2e9f-4fb9-bfa7-c5dc31dd586a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 18:10:52.252024 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:10:52.251993    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0353158b-2e9f-4fb9-bfa7-c5dc31dd586a-kube-api-access-jzdrd" (OuterVolumeSpecName: "kube-api-access-jzdrd") pod "0353158b-2e9f-4fb9-bfa7-c5dc31dd586a" (UID: "0353158b-2e9f-4fb9-bfa7-c5dc31dd586a"). InnerVolumeSpecName "kube-api-access-jzdrd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 18:10:52.252128 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:10:52.252025    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0353158b-2e9f-4fb9-bfa7-c5dc31dd586a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0353158b-2e9f-4fb9-bfa7-c5dc31dd586a" (UID: "0353158b-2e9f-4fb9-bfa7-c5dc31dd586a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 18:10:52.350956 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:10:52.350908    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzdrd\" (UniqueName: \"kubernetes.io/projected/0353158b-2e9f-4fb9-bfa7-c5dc31dd586a-kube-api-access-jzdrd\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 18:10:52.350956 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:10:52.350951    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0353158b-2e9f-4fb9-bfa7-c5dc31dd586a-kserve-provision-location\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 18:10:52.350956 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:10:52.350962    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0353158b-2e9f-4fb9-bfa7-c5dc31dd586a-cabundle-cert\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 18:10:52.350956 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:10:52.350971    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0353158b-2e9f-4fb9-bfa7-c5dc31dd586a-proxy-tls\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 18:10:52.351248 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:10:52.350979    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0353158b-2e9f-4fb9-bfa7-c5dc31dd586a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 18:10:52.627602 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:10:52.627571    2555 generic.go:356] "Generic (PLEG): container finished" podID="0353158b-2e9f-4fb9-bfa7-c5dc31dd586a" containerID="7086c27cbf3eaa4a85661c4e250e08becc65d5ff64af2d239353e43fa7ee0bdf" exitCode=0
Jul 30 18:10:52.627785 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:10:52.627629    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-svrxw"
Jul 30 18:10:52.627785 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:10:52.627654    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-svrxw" event={"ID":"0353158b-2e9f-4fb9-bfa7-c5dc31dd586a","Type":"ContainerDied","Data":"7086c27cbf3eaa4a85661c4e250e08becc65d5ff64af2d239353e43fa7ee0bdf"}
Jul 30 18:10:52.627785 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:10:52.627714    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-svrxw" event={"ID":"0353158b-2e9f-4fb9-bfa7-c5dc31dd586a","Type":"ContainerDied","Data":"b082e856e0831d7580c0dc0a38bd477ff98f0325768c87a9e568044000f2bc8d"}
Jul 30 18:10:52.627785 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:10:52.627741    2555 scope.go:117] "RemoveContainer" containerID="86b98969428f56d95a60ae8b43f0e1e7b53f648025b3b77fe83a027defc17372"
Jul 30 18:10:52.637047 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:10:52.637031    2555 scope.go:117] "RemoveContainer" containerID="7086c27cbf3eaa4a85661c4e250e08becc65d5ff64af2d239353e43fa7ee0bdf"
Jul 30 18:10:52.644581 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:10:52.644565    2555 scope.go:117] "RemoveContainer" containerID="88e39645121230431db938b5886846365896b5e8563cb5f7c8217606f83203e3"
Jul 30 18:10:52.652336 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:10:52.652316    2555 scope.go:117] "RemoveContainer" containerID="86b98969428f56d95a60ae8b43f0e1e7b53f648025b3b77fe83a027defc17372"
Jul 30 18:10:52.652713 ip-10-0-143-190 kubenswrapper[2555]: E0730 18:10:52.652678    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86b98969428f56d95a60ae8b43f0e1e7b53f648025b3b77fe83a027defc17372\": container with ID starting with 86b98969428f56d95a60ae8b43f0e1e7b53f648025b3b77fe83a027defc17372 not found: ID does not exist" containerID="86b98969428f56d95a60ae8b43f0e1e7b53f648025b3b77fe83a027defc17372"
Jul 30 18:10:52.652816 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:10:52.652717    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86b98969428f56d95a60ae8b43f0e1e7b53f648025b3b77fe83a027defc17372"} err="failed to get container status \"86b98969428f56d95a60ae8b43f0e1e7b53f648025b3b77fe83a027defc17372\": rpc error: code = NotFound desc = could not find container \"86b98969428f56d95a60ae8b43f0e1e7b53f648025b3b77fe83a027defc17372\": container with ID starting with 86b98969428f56d95a60ae8b43f0e1e7b53f648025b3b77fe83a027defc17372 not found: ID does not exist"
Jul 30 18:10:52.652816 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:10:52.652735    2555 scope.go:117] "RemoveContainer" containerID="7086c27cbf3eaa4a85661c4e250e08becc65d5ff64af2d239353e43fa7ee0bdf"
Jul 30 18:10:52.653128 ip-10-0-143-190 kubenswrapper[2555]: E0730 18:10:52.652980    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7086c27cbf3eaa4a85661c4e250e08becc65d5ff64af2d239353e43fa7ee0bdf\": container with ID starting with 7086c27cbf3eaa4a85661c4e250e08becc65d5ff64af2d239353e43fa7ee0bdf not found: ID does not exist" containerID="7086c27cbf3eaa4a85661c4e250e08becc65d5ff64af2d239353e43fa7ee0bdf"
Jul 30 18:10:52.653128 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:10:52.653038    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7086c27cbf3eaa4a85661c4e250e08becc65d5ff64af2d239353e43fa7ee0bdf"} err="failed to get container status \"7086c27cbf3eaa4a85661c4e250e08becc65d5ff64af2d239353e43fa7ee0bdf\": rpc error: code = NotFound desc = could not find container \"7086c27cbf3eaa4a85661c4e250e08becc65d5ff64af2d239353e43fa7ee0bdf\": container with ID starting with 7086c27cbf3eaa4a85661c4e250e08becc65d5ff64af2d239353e43fa7ee0bdf not found: ID does not exist"
Jul 30 18:10:52.653128 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:10:52.653066    2555 scope.go:117] "RemoveContainer" containerID="88e39645121230431db938b5886846365896b5e8563cb5f7c8217606f83203e3"
Jul 30 18:10:52.653346 ip-10-0-143-190 kubenswrapper[2555]: E0730 18:10:52.653318    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88e39645121230431db938b5886846365896b5e8563cb5f7c8217606f83203e3\": container with ID starting with 88e39645121230431db938b5886846365896b5e8563cb5f7c8217606f83203e3 not found: ID does not exist" containerID="88e39645121230431db938b5886846365896b5e8563cb5f7c8217606f83203e3"
Jul 30 18:10:52.653397 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:10:52.653352    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88e39645121230431db938b5886846365896b5e8563cb5f7c8217606f83203e3"} err="failed to get container status \"88e39645121230431db938b5886846365896b5e8563cb5f7c8217606f83203e3\": rpc error: code = NotFound desc = could not find container \"88e39645121230431db938b5886846365896b5e8563cb5f7c8217606f83203e3\": container with ID starting with 88e39645121230431db938b5886846365896b5e8563cb5f7c8217606f83203e3 not found: ID does not exist"
Jul 30 18:10:52.654328 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:10:52.654309    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-svrxw"]
Jul 30 18:10:52.657831 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:10:52.657811    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-svrxw"]
Jul 30 18:10:54.060311 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:10:54.060282    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0353158b-2e9f-4fb9-bfa7-c5dc31dd586a" path="/var/lib/kubelet/pods/0353158b-2e9f-4fb9-bfa7-c5dc31dd586a/volumes"
Jul 30 18:11:17.617151 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:11:17.617117    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:12:20.440659 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:12:20.440577    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:12:25.537804 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:12:25.537765    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:12:25.846034 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:12:25.845986    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:13:46.856786 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:13:46.856745    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:13:46.914760 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:13:46.914728    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:13:55.719072 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:13:55.718967    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:14:00.823179 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:14:00.823139    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:14:01.297783 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:14:01.297747    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:14:41.687043 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:14:41.686990    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:14:46.796692 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:14:46.796655    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:14:47.008357 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:14:47.008304    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:15:11.685801 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:15:11.685756    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:15:27.995523 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:15:27.995426    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:15:33.119579 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:15:33.119547    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:15:33.146789 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:15:33.146753    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-b77nd"]
Jul 30 18:15:33.147227 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:15:33.147209    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0353158b-2e9f-4fb9-bfa7-c5dc31dd586a" containerName="kserve-container"
Jul 30 18:15:33.147227 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:15:33.147229    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="0353158b-2e9f-4fb9-bfa7-c5dc31dd586a" containerName="kserve-container"
Jul 30 18:15:33.147396 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:15:33.147260    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0353158b-2e9f-4fb9-bfa7-c5dc31dd586a" containerName="kube-rbac-proxy"
Jul 30 18:15:33.147396 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:15:33.147269    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="0353158b-2e9f-4fb9-bfa7-c5dc31dd586a" containerName="kube-rbac-proxy"
Jul 30 18:15:33.147396 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:15:33.147296    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0353158b-2e9f-4fb9-bfa7-c5dc31dd586a" containerName="storage-initializer"
Jul 30 18:15:33.147396 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:15:33.147305    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="0353158b-2e9f-4fb9-bfa7-c5dc31dd586a" containerName="storage-initializer"
Jul 30 18:15:33.147396 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:15:33.147390    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="0353158b-2e9f-4fb9-bfa7-c5dc31dd586a" containerName="kserve-container"
Jul 30 18:15:33.147651 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:15:33.147405    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="0353158b-2e9f-4fb9-bfa7-c5dc31dd586a" containerName="kube-rbac-proxy"
Jul 30 18:15:33.150651 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:15:33.150629    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-b77nd"
Jul 30 18:15:33.153329 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:15:33.153309    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-predictor-serving-cert\""
Jul 30 18:15:33.153419 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:15:33.153349    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 30 18:15:33.154648 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:15:33.154629    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 30 18:15:33.154749 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:15:33.154630    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 30 18:15:33.154749 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:15:33.154685    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-kube-rbac-proxy-sar-config\""
Jul 30 18:15:33.154749 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:15:33.154707    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 30 18:15:33.154749 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:15:33.154694    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-plw5t\""
Jul 30 18:15:33.162442 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:15:33.162419    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-b77nd"]
Jul 30 18:15:33.213719 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:15:33.213686    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:15:33.268485 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:15:33.268456    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e89e467-d102-4859-9404-2d6be6c62ad8-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-b77nd\" (UID: \"6e89e467-d102-4859-9404-2d6be6c62ad8\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-b77nd"
Jul 30 18:15:33.268668 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:15:33.268500    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e89e467-d102-4859-9404-2d6be6c62ad8-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-b77nd\" (UID: \"6e89e467-d102-4859-9404-2d6be6c62ad8\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-b77nd"
Jul 30 18:15:33.268668 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:15:33.268622    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e89e467-d102-4859-9404-2d6be6c62ad8-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-b77nd\" (UID: \"6e89e467-d102-4859-9404-2d6be6c62ad8\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-b77nd"
Jul 30 18:15:33.268762 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:15:33.268667    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e89e467-d102-4859-9404-2d6be6c62ad8-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-b77nd\" (UID: \"6e89e467-d102-4859-9404-2d6be6c62ad8\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-b77nd"
Jul 30 18:15:33.268762 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:15:33.268744    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zn9b\" (UniqueName: \"kubernetes.io/projected/6e89e467-d102-4859-9404-2d6be6c62ad8-kube-api-access-9zn9b\") pod \"isvc-triton-predictor-5d7db58676-b77nd\" (UID: \"6e89e467-d102-4859-9404-2d6be6c62ad8\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-b77nd"
Jul 30 18:15:33.369665 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:15:33.369557    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e89e467-d102-4859-9404-2d6be6c62ad8-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-b77nd\" (UID: \"6e89e467-d102-4859-9404-2d6be6c62ad8\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-b77nd"
Jul 30 18:15:33.369665 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:15:33.369611    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e89e467-d102-4859-9404-2d6be6c62ad8-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-b77nd\" (UID: \"6e89e467-d102-4859-9404-2d6be6c62ad8\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-b77nd"
Jul 30 18:15:33.369665 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:15:33.369653    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e89e467-d102-4859-9404-2d6be6c62ad8-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-b77nd\" (UID: \"6e89e467-d102-4859-9404-2d6be6c62ad8\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-b77nd"
Jul 30 18:15:33.369980 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:15:33.369770    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e89e467-d102-4859-9404-2d6be6c62ad8-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-b77nd\" (UID: \"6e89e467-d102-4859-9404-2d6be6c62ad8\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-b77nd"
Jul 30 18:15:33.369980 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:15:33.369832    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zn9b\" (UniqueName: \"kubernetes.io/projected/6e89e467-d102-4859-9404-2d6be6c62ad8-kube-api-access-9zn9b\") pod \"isvc-triton-predictor-5d7db58676-b77nd\" (UID: \"6e89e467-d102-4859-9404-2d6be6c62ad8\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-b77nd"
Jul 30 18:15:33.370171 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:15:33.370146    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e89e467-d102-4859-9404-2d6be6c62ad8-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-b77nd\" (UID: \"6e89e467-d102-4859-9404-2d6be6c62ad8\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-b77nd"
Jul 30 18:15:33.370453 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:15:33.370430    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e89e467-d102-4859-9404-2d6be6c62ad8-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-b77nd\" (UID: \"6e89e467-d102-4859-9404-2d6be6c62ad8\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-b77nd"
Jul 30 18:15:33.370496 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:15:33.370437    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e89e467-d102-4859-9404-2d6be6c62ad8-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-b77nd\" (UID: \"6e89e467-d102-4859-9404-2d6be6c62ad8\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-b77nd"
Jul 30 18:15:33.372109 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:15:33.372092    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e89e467-d102-4859-9404-2d6be6c62ad8-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-b77nd\" (UID: \"6e89e467-d102-4859-9404-2d6be6c62ad8\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-b77nd"
Jul 30 18:15:33.379516 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:15:33.379492    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zn9b\" (UniqueName: \"kubernetes.io/projected/6e89e467-d102-4859-9404-2d6be6c62ad8-kube-api-access-9zn9b\") pod \"isvc-triton-predictor-5d7db58676-b77nd\" (UID: \"6e89e467-d102-4859-9404-2d6be6c62ad8\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-b77nd"
Jul 30 18:15:33.462700 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:15:33.462643    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-b77nd"
Jul 30 18:15:33.595558 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:15:33.595536    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-b77nd"]
Jul 30 18:15:33.597943 ip-10-0-143-190 kubenswrapper[2555]: W0730 18:15:33.597915    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e89e467_d102_4859_9404_2d6be6c62ad8.slice/crio-0bb182f728bf612e38fcddd56e8c2e8a3cef463d215d2bc8c3c0e1970cbe4d19 WatchSource:0}: Error finding container 0bb182f728bf612e38fcddd56e8c2e8a3cef463d215d2bc8c3c0e1970cbe4d19: Status 404 returned error can't find the container with id 0bb182f728bf612e38fcddd56e8c2e8a3cef463d215d2bc8c3c0e1970cbe4d19
Jul 30 18:15:33.919833 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:15:33.919742    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-b77nd" event={"ID":"6e89e467-d102-4859-9404-2d6be6c62ad8","Type":"ContainerStarted","Data":"4ef00dcc9da60b43e416902f73ea4bdf8c8b28fa24f1f32cdce26113c6ded5c9"}
Jul 30 18:15:33.919833 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:15:33.919801    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-b77nd" event={"ID":"6e89e467-d102-4859-9404-2d6be6c62ad8","Type":"ContainerStarted","Data":"0bb182f728bf612e38fcddd56e8c2e8a3cef463d215d2bc8c3c0e1970cbe4d19"}
Jul 30 18:15:37.939001 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:15:37.938967    2555 generic.go:356] "Generic (PLEG): container finished" podID="6e89e467-d102-4859-9404-2d6be6c62ad8" containerID="4ef00dcc9da60b43e416902f73ea4bdf8c8b28fa24f1f32cdce26113c6ded5c9" exitCode=0
Jul 30 18:15:37.939463 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:15:37.939038    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-b77nd" event={"ID":"6e89e467-d102-4859-9404-2d6be6c62ad8","Type":"ContainerDied","Data":"4ef00dcc9da60b43e416902f73ea4bdf8c8b28fa24f1f32cdce26113c6ded5c9"}
Jul 30 18:15:37.942712 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:15:37.942691    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 30 18:16:47.679029 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:16:47.678844    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:17:32.626617 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:17:32.626581    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-b77nd" event={"ID":"6e89e467-d102-4859-9404-2d6be6c62ad8","Type":"ContainerStarted","Data":"bd96731a896de57875a5e2ce14ed7c63635be0dab75eaf6c4017423c741622c0"}
Jul 30 18:17:32.627113 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:17:32.626650    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-b77nd" event={"ID":"6e89e467-d102-4859-9404-2d6be6c62ad8","Type":"ContainerStarted","Data":"faeb8afbd1cebb6a0229a9082af7e93c489dd4ba71e3b07a5eee0d8acb1679bc"}
Jul 30 18:17:32.627113 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:17:32.626718    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-b77nd"
Jul 30 18:17:32.653215 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:17:32.653167    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-b77nd" podStartSLOduration=1.6409563980000001 podStartE2EDuration="1m59.653153752s" totalImagesPullingTime="1m54.159531133s" totalInitContainerRuntime="3.852666221s" isStatefulPod=false podCreationTimestamp="2026-07-30 18:15:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 18:17:32.652514672 +0000 UTC m=+4627.139882076" watchObservedRunningTime="2026-07-30 18:17:32.653153752 +0000 UTC m=+4627.140521153"
Jul 30 18:17:33.630985 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:17:33.630954    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-b77nd"
Jul 30 18:17:33.632457 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:17:33.632426    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-b77nd" podUID="6e89e467-d102-4859-9404-2d6be6c62ad8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Jul 30 18:17:34.635505 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:17:34.635465    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-b77nd" podUID="6e89e467-d102-4859-9404-2d6be6c62ad8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Jul 30 18:17:39.640484 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:17:39.640452    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-b77nd"
Jul 30 18:17:39.641586 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:17:39.641567    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-b77nd"
Jul 30 18:17:44.691318 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:17:44.691284    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-b77nd"]
Jul 30 18:17:44.691808 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:17:44.691550    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-b77nd" podUID="6e89e467-d102-4859-9404-2d6be6c62ad8" containerName="kserve-container" containerID="cri-o://faeb8afbd1cebb6a0229a9082af7e93c489dd4ba71e3b07a5eee0d8acb1679bc" gracePeriod=30
Jul 30 18:17:44.691808 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:17:44.691594    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-b77nd" podUID="6e89e467-d102-4859-9404-2d6be6c62ad8" containerName="kube-rbac-proxy" containerID="cri-o://bd96731a896de57875a5e2ce14ed7c63635be0dab75eaf6c4017423c741622c0" gracePeriod=30
Jul 30 18:17:44.714087 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:17:44.714024    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:17:45.689703 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:17:45.689673    2555 generic.go:356] "Generic (PLEG): container finished" podID="6e89e467-d102-4859-9404-2d6be6c62ad8" containerID="bd96731a896de57875a5e2ce14ed7c63635be0dab75eaf6c4017423c741622c0" exitCode=2
Jul 30 18:17:45.689960 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:17:45.689746    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-b77nd" event={"ID":"6e89e467-d102-4859-9404-2d6be6c62ad8","Type":"ContainerDied","Data":"bd96731a896de57875a5e2ce14ed7c63635be0dab75eaf6c4017423c741622c0"}
Jul 30 18:17:47.702474 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:17:47.702440    2555 generic.go:356] "Generic (PLEG): container finished" podID="6e89e467-d102-4859-9404-2d6be6c62ad8" containerID="faeb8afbd1cebb6a0229a9082af7e93c489dd4ba71e3b07a5eee0d8acb1679bc" exitCode=0
Jul 30 18:17:47.702843 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:17:47.702506    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-b77nd" event={"ID":"6e89e467-d102-4859-9404-2d6be6c62ad8","Type":"ContainerDied","Data":"faeb8afbd1cebb6a0229a9082af7e93c489dd4ba71e3b07a5eee0d8acb1679bc"}
Jul 30 18:17:47.763305 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:17:47.763245    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-b77nd"
Jul 30 18:17:47.878084 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:17:47.878051    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9zn9b\" (UniqueName: \"kubernetes.io/projected/6e89e467-d102-4859-9404-2d6be6c62ad8-kube-api-access-9zn9b\") pod \"6e89e467-d102-4859-9404-2d6be6c62ad8\" (UID: \"6e89e467-d102-4859-9404-2d6be6c62ad8\") "
Jul 30 18:17:47.878288 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:17:47.878132    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e89e467-d102-4859-9404-2d6be6c62ad8-proxy-tls\") pod \"6e89e467-d102-4859-9404-2d6be6c62ad8\" (UID: \"6e89e467-d102-4859-9404-2d6be6c62ad8\") "
Jul 30 18:17:47.878288 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:17:47.878235    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e89e467-d102-4859-9404-2d6be6c62ad8-kserve-provision-location\") pod \"6e89e467-d102-4859-9404-2d6be6c62ad8\" (UID: \"6e89e467-d102-4859-9404-2d6be6c62ad8\") "
Jul 30 18:17:47.878415 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:17:47.878292    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e89e467-d102-4859-9404-2d6be6c62ad8-kube-rbac-proxy-sar-config\") pod \"6e89e467-d102-4859-9404-2d6be6c62ad8\" (UID: \"6e89e467-d102-4859-9404-2d6be6c62ad8\") "
Jul 30 18:17:47.878415 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:17:47.878322    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e89e467-d102-4859-9404-2d6be6c62ad8-cabundle-cert\") pod \"6e89e467-d102-4859-9404-2d6be6c62ad8\" (UID: \"6e89e467-d102-4859-9404-2d6be6c62ad8\") "
Jul 30 18:17:47.878682 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:17:47.878654    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e89e467-d102-4859-9404-2d6be6c62ad8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6e89e467-d102-4859-9404-2d6be6c62ad8" (UID: "6e89e467-d102-4859-9404-2d6be6c62ad8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 18:17:47.878682 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:17:47.878672    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e89e467-d102-4859-9404-2d6be6c62ad8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6e89e467-d102-4859-9404-2d6be6c62ad8" (UID: "6e89e467-d102-4859-9404-2d6be6c62ad8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 18:17:47.880183 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:17:47.880157    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e89e467-d102-4859-9404-2d6be6c62ad8-kube-api-access-9zn9b" (OuterVolumeSpecName: "kube-api-access-9zn9b") pod "6e89e467-d102-4859-9404-2d6be6c62ad8" (UID: "6e89e467-d102-4859-9404-2d6be6c62ad8"). InnerVolumeSpecName "kube-api-access-9zn9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 18:17:47.880290 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:17:47.880256    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e89e467-d102-4859-9404-2d6be6c62ad8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6e89e467-d102-4859-9404-2d6be6c62ad8" (UID: "6e89e467-d102-4859-9404-2d6be6c62ad8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 18:17:47.885792 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:17:47.885772    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e89e467-d102-4859-9404-2d6be6c62ad8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6e89e467-d102-4859-9404-2d6be6c62ad8" (UID: "6e89e467-d102-4859-9404-2d6be6c62ad8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 18:17:47.979447 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:17:47.979414    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e89e467-d102-4859-9404-2d6be6c62ad8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 18:17:47.979447 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:17:47.979440    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e89e467-d102-4859-9404-2d6be6c62ad8-cabundle-cert\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 18:17:47.979447 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:17:47.979450    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zn9b\" (UniqueName: \"kubernetes.io/projected/6e89e467-d102-4859-9404-2d6be6c62ad8-kube-api-access-9zn9b\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 18:17:47.979676 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:17:47.979462    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e89e467-d102-4859-9404-2d6be6c62ad8-proxy-tls\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 18:17:47.979676 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:17:47.979471    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e89e467-d102-4859-9404-2d6be6c62ad8-kserve-provision-location\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 18:17:48.709627 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:17:48.709600    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-b77nd"
Jul 30 18:17:48.710095 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:17:48.709595    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-b77nd" event={"ID":"6e89e467-d102-4859-9404-2d6be6c62ad8","Type":"ContainerDied","Data":"0bb182f728bf612e38fcddd56e8c2e8a3cef463d215d2bc8c3c0e1970cbe4d19"}
Jul 30 18:17:48.710095 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:17:48.709732    2555 scope.go:117] "RemoveContainer" containerID="bd96731a896de57875a5e2ce14ed7c63635be0dab75eaf6c4017423c741622c0"
Jul 30 18:17:48.718134 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:17:48.718111    2555 scope.go:117] "RemoveContainer" containerID="faeb8afbd1cebb6a0229a9082af7e93c489dd4ba71e3b07a5eee0d8acb1679bc"
Jul 30 18:17:48.725660 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:17:48.725640    2555 scope.go:117] "RemoveContainer" containerID="4ef00dcc9da60b43e416902f73ea4bdf8c8b28fa24f1f32cdce26113c6ded5c9"
Jul 30 18:17:48.731995 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:17:48.731971    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-b77nd"]
Jul 30 18:17:48.736163 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:17:48.736141    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-b77nd"]
Jul 30 18:17:49.782174 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:17:49.782141    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:17:50.060726 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:17:50.060691    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e89e467-d102-4859-9404-2d6be6c62ad8" path="/var/lib/kubelet/pods/6e89e467-d102-4859-9404-2d6be6c62ad8/volumes"
Jul 30 18:17:50.085087 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:17:50.085056    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:19:30.010741 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:19:30.010707    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:19:35.080432 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:19:35.080386    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:19:35.138794 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:19:35.138764    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-shb8f"]
Jul 30 18:19:35.139167 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:19:35.139153    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e89e467-d102-4859-9404-2d6be6c62ad8" containerName="kube-rbac-proxy"
Jul 30 18:19:35.139216 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:19:35.139169    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e89e467-d102-4859-9404-2d6be6c62ad8" containerName="kube-rbac-proxy"
Jul 30 18:19:35.139216 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:19:35.139192    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e89e467-d102-4859-9404-2d6be6c62ad8" containerName="storage-initializer"
Jul 30 18:19:35.139216 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:19:35.139198    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e89e467-d102-4859-9404-2d6be6c62ad8" containerName="storage-initializer"
Jul 30 18:19:35.139216 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:19:35.139204    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e89e467-d102-4859-9404-2d6be6c62ad8" containerName="kserve-container"
Jul 30 18:19:35.139216 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:19:35.139209    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e89e467-d102-4859-9404-2d6be6c62ad8" containerName="kserve-container"
Jul 30 18:19:35.139378 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:19:35.139266    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e89e467-d102-4859-9404-2d6be6c62ad8" containerName="kserve-container"
Jul 30 18:19:35.139378 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:19:35.139273    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e89e467-d102-4859-9404-2d6be6c62ad8" containerName="kube-rbac-proxy"
Jul 30 18:19:35.141750 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:19:35.141735    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-shb8f"
Jul 30 18:19:35.144294 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:19:35.144270    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 30 18:19:35.144420 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:19:35.144384    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 30 18:19:35.144489 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:19:35.144427    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 30 18:19:35.144489 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:19:35.144477    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 30 18:19:35.144793 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:19:35.144779    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Jul 30 18:19:35.145538 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:19:35.145518    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-predictor-serving-cert\""
Jul 30 18:19:35.145654 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:19:35.145542    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-plw5t\""
Jul 30 18:19:35.152788 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:19:35.152769    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-shb8f"]
Jul 30 18:19:35.215297 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:19:35.215266    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f59dedb6-adb5-409a-bbc6-c487b9426a0b-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-shb8f\" (UID: \"f59dedb6-adb5-409a-bbc6-c487b9426a0b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-shb8f"
Jul 30 18:19:35.215474 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:19:35.215311    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f59dedb6-adb5-409a-bbc6-c487b9426a0b-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-shb8f\" (UID: \"f59dedb6-adb5-409a-bbc6-c487b9426a0b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-shb8f"
Jul 30 18:19:35.215474 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:19:35.215367    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f59dedb6-adb5-409a-bbc6-c487b9426a0b-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-shb8f\" (UID: \"f59dedb6-adb5-409a-bbc6-c487b9426a0b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-shb8f"
Jul 30 18:19:35.215474 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:19:35.215453    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdrlt\" (UniqueName: \"kubernetes.io/projected/f59dedb6-adb5-409a-bbc6-c487b9426a0b-kube-api-access-pdrlt\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-shb8f\" (UID: \"f59dedb6-adb5-409a-bbc6-c487b9426a0b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-shb8f"
Jul 30 18:19:35.215600 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:19:35.215480    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f59dedb6-adb5-409a-bbc6-c487b9426a0b-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-shb8f\" (UID: \"f59dedb6-adb5-409a-bbc6-c487b9426a0b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-shb8f"
Jul 30 18:19:35.316180 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:19:35.316141    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f59dedb6-adb5-409a-bbc6-c487b9426a0b-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-shb8f\" (UID: \"f59dedb6-adb5-409a-bbc6-c487b9426a0b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-shb8f"
Jul 30 18:19:35.316471 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:19:35.316199    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f59dedb6-adb5-409a-bbc6-c487b9426a0b-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-shb8f\" (UID: \"f59dedb6-adb5-409a-bbc6-c487b9426a0b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-shb8f"
Jul 30 18:19:35.316471 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:19:35.316248    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f59dedb6-adb5-409a-bbc6-c487b9426a0b-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-shb8f\" (UID: \"f59dedb6-adb5-409a-bbc6-c487b9426a0b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-shb8f"
Jul 30 18:19:35.316471 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:19:35.316307    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdrlt\" (UniqueName: \"kubernetes.io/projected/f59dedb6-adb5-409a-bbc6-c487b9426a0b-kube-api-access-pdrlt\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-shb8f\" (UID: \"f59dedb6-adb5-409a-bbc6-c487b9426a0b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-shb8f"
Jul 30 18:19:35.316471 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:19:35.316330    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f59dedb6-adb5-409a-bbc6-c487b9426a0b-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-shb8f\" (UID: \"f59dedb6-adb5-409a-bbc6-c487b9426a0b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-shb8f"
Jul 30 18:19:35.316729 ip-10-0-143-190 kubenswrapper[2555]: E0730 18:19:35.316527    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-serving-cert: secret "isvc-xgboost-v2-mlserver-predictor-serving-cert" not found
Jul 30 18:19:35.316729 ip-10-0-143-190 kubenswrapper[2555]: E0730 18:19:35.316615    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f59dedb6-adb5-409a-bbc6-c487b9426a0b-proxy-tls podName:f59dedb6-adb5-409a-bbc6-c487b9426a0b nodeName:}" failed. No retries permitted until 2026-07-30 18:19:35.816591964 +0000 UTC m=+4750.303959355 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f59dedb6-adb5-409a-bbc6-c487b9426a0b-proxy-tls") pod "isvc-xgboost-v2-mlserver-predictor-7b55b84f54-shb8f" (UID: "f59dedb6-adb5-409a-bbc6-c487b9426a0b") : secret "isvc-xgboost-v2-mlserver-predictor-serving-cert" not found
Jul 30 18:19:35.316833 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:19:35.316746    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f59dedb6-adb5-409a-bbc6-c487b9426a0b-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-shb8f\" (UID: \"f59dedb6-adb5-409a-bbc6-c487b9426a0b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-shb8f"
Jul 30 18:19:35.316873 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:19:35.316852    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f59dedb6-adb5-409a-bbc6-c487b9426a0b-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-shb8f\" (UID: \"f59dedb6-adb5-409a-bbc6-c487b9426a0b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-shb8f"
Jul 30 18:19:35.317034 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:19:35.316989    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f59dedb6-adb5-409a-bbc6-c487b9426a0b-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-shb8f\" (UID: \"f59dedb6-adb5-409a-bbc6-c487b9426a0b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-shb8f"
Jul 30 18:19:35.325937 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:19:35.325913    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdrlt\" (UniqueName: \"kubernetes.io/projected/f59dedb6-adb5-409a-bbc6-c487b9426a0b-kube-api-access-pdrlt\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-shb8f\" (UID: \"f59dedb6-adb5-409a-bbc6-c487b9426a0b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-shb8f"
Jul 30 18:19:35.398220 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:19:35.398141    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:19:35.820798 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:19:35.820766    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f59dedb6-adb5-409a-bbc6-c487b9426a0b-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-shb8f\" (UID: \"f59dedb6-adb5-409a-bbc6-c487b9426a0b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-shb8f"
Jul 30 18:19:35.823187 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:19:35.823157    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f59dedb6-adb5-409a-bbc6-c487b9426a0b-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-shb8f\" (UID: \"f59dedb6-adb5-409a-bbc6-c487b9426a0b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-shb8f"
Jul 30 18:19:36.052442 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:19:36.052409    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-shb8f"
Jul 30 18:19:36.186767 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:19:36.186740    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-shb8f"]
Jul 30 18:19:36.189270 ip-10-0-143-190 kubenswrapper[2555]: W0730 18:19:36.189244    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf59dedb6_adb5_409a_bbc6_c487b9426a0b.slice/crio-0ef46c6af47c25d1aaf4ff688198a133989f9ae0a5d1e03c15df7451510dbbf0 WatchSource:0}: Error finding container 0ef46c6af47c25d1aaf4ff688198a133989f9ae0a5d1e03c15df7451510dbbf0: Status 404 returned error can't find the container with id 0ef46c6af47c25d1aaf4ff688198a133989f9ae0a5d1e03c15df7451510dbbf0
Jul 30 18:19:36.202571 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:19:36.202547    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-shb8f" event={"ID":"f59dedb6-adb5-409a-bbc6-c487b9426a0b","Type":"ContainerStarted","Data":"0ef46c6af47c25d1aaf4ff688198a133989f9ae0a5d1e03c15df7451510dbbf0"}
Jul 30 18:19:37.208919 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:19:37.208875    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-shb8f" event={"ID":"f59dedb6-adb5-409a-bbc6-c487b9426a0b","Type":"ContainerStarted","Data":"2796cc533f348744b3d1fb26293feb2818d747981a75e3c644f5efc361d04acb"}
Jul 30 18:19:41.227554 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:19:41.227522    2555 generic.go:356] "Generic (PLEG): container finished" podID="f59dedb6-adb5-409a-bbc6-c487b9426a0b" containerID="2796cc533f348744b3d1fb26293feb2818d747981a75e3c644f5efc361d04acb" exitCode=0
Jul 30 18:19:41.227944 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:19:41.227594    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-shb8f" event={"ID":"f59dedb6-adb5-409a-bbc6-c487b9426a0b","Type":"ContainerDied","Data":"2796cc533f348744b3d1fb26293feb2818d747981a75e3c644f5efc361d04acb"}
Jul 30 18:19:42.233868 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:19:42.233835    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-shb8f" event={"ID":"f59dedb6-adb5-409a-bbc6-c487b9426a0b","Type":"ContainerStarted","Data":"d82dbdad04287591b623f851f4218ee52fc2f6038dd58cc1f47d5ecbee0ea7c0"}
Jul 30 18:19:42.234283 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:19:42.233901    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-shb8f" event={"ID":"f59dedb6-adb5-409a-bbc6-c487b9426a0b","Type":"ContainerStarted","Data":"18f990bded7efa9254cbb95d6ad55087748a00282165330189accefb3ca2561f"}
Jul 30 18:19:42.256600 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:19:42.256549    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-shb8f" podStartSLOduration=3.262863965 podStartE2EDuration="7.256533389s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.993669424s" isStatefulPod=false podCreationTimestamp="2026-07-30 18:19:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 18:19:42.254594904 +0000 UTC m=+4756.741962306" watchObservedRunningTime="2026-07-30 18:19:42.256533389 +0000 UTC m=+4756.743900790"
Jul 30 18:19:47.234190 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:19:47.234155    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-shb8f"
Jul 30 18:19:47.234658 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:19:47.234545    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-shb8f"
Jul 30 18:19:47.240780 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:19:47.240757    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-shb8f"
Jul 30 18:20:16.632519 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:20:16.632485    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:20:17.973272 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:20:17.973240    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:20:18.302693 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:20:18.302661    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-shb8f"
Jul 30 18:20:25.219984 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:20:25.219952    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-shb8f"]
Jul 30 18:20:25.220402 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:20:25.220207    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-shb8f" podUID="f59dedb6-adb5-409a-bbc6-c487b9426a0b" containerName="kserve-container" containerID="cri-o://18f990bded7efa9254cbb95d6ad55087748a00282165330189accefb3ca2561f" gracePeriod=30
Jul 30 18:20:25.220402 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:20:25.220232    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-shb8f" podUID="f59dedb6-adb5-409a-bbc6-c487b9426a0b" containerName="kube-rbac-proxy" containerID="cri-o://d82dbdad04287591b623f851f4218ee52fc2f6038dd58cc1f47d5ecbee0ea7c0" gracePeriod=30
Jul 30 18:20:25.242855 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:20:25.242826    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:20:25.433682 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:20:25.433645    2555 generic.go:356] "Generic (PLEG): container finished" podID="f59dedb6-adb5-409a-bbc6-c487b9426a0b" containerID="d82dbdad04287591b623f851f4218ee52fc2f6038dd58cc1f47d5ecbee0ea7c0" exitCode=2
Jul 30 18:20:25.433851 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:20:25.433722    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-shb8f" event={"ID":"f59dedb6-adb5-409a-bbc6-c487b9426a0b","Type":"ContainerDied","Data":"d82dbdad04287591b623f851f4218ee52fc2f6038dd58cc1f47d5ecbee0ea7c0"}
Jul 30 18:20:27.241074 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:20:27.241002    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-shb8f" podUID="f59dedb6-adb5-409a-bbc6-c487b9426a0b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.51:8643/healthz\": dial tcp 10.133.0.51:8643: connect: connection refused"
Jul 30 18:20:28.260496 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:20:28.260456    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-shb8f" podUID="f59dedb6-adb5-409a-bbc6-c487b9426a0b" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.51:8080/v2/models/isvc-xgboost-v2-mlserver/ready\": dial tcp 10.133.0.51:8080: connect: connection refused"
Jul 30 18:20:30.368124 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:20:30.368075    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:20:30.398574 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:20:30.398541    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-flzkd"]
Jul 30 18:20:30.401587 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:20:30.401558    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-flzkd"
Jul 30 18:20:30.403999 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:20:30.403979    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-predictor-serving-cert\""
Jul 30 18:20:30.403999 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:20:30.403977    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Jul 30 18:20:30.416462 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:20:30.416218    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-flzkd"]
Jul 30 18:20:30.485316 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:20:30.485279    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a087c70-a94f-45ee-8469-e19cac8100a4-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-flzkd\" (UID: \"3a087c70-a94f-45ee-8469-e19cac8100a4\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-flzkd"
Jul 30 18:20:30.485495 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:20:30.485322    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbq96\" (UniqueName: \"kubernetes.io/projected/3a087c70-a94f-45ee-8469-e19cac8100a4-kube-api-access-tbq96\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-flzkd\" (UID: \"3a087c70-a94f-45ee-8469-e19cac8100a4\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-flzkd"
Jul 30 18:20:30.485495 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:20:30.485393    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a087c70-a94f-45ee-8469-e19cac8100a4-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-flzkd\" (UID: \"3a087c70-a94f-45ee-8469-e19cac8100a4\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-flzkd"
Jul 30 18:20:30.485495 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:20:30.485448    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a087c70-a94f-45ee-8469-e19cac8100a4-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-flzkd\" (UID: \"3a087c70-a94f-45ee-8469-e19cac8100a4\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-flzkd"
Jul 30 18:20:30.485495 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:20:30.485483    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a087c70-a94f-45ee-8469-e19cac8100a4-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-flzkd\" (UID: \"3a087c70-a94f-45ee-8469-e19cac8100a4\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-flzkd"
Jul 30 18:20:30.586296 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:20:30.586246    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a087c70-a94f-45ee-8469-e19cac8100a4-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-flzkd\" (UID: \"3a087c70-a94f-45ee-8469-e19cac8100a4\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-flzkd"
Jul 30 18:20:30.586296 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:20:30.586301    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbq96\" (UniqueName: \"kubernetes.io/projected/3a087c70-a94f-45ee-8469-e19cac8100a4-kube-api-access-tbq96\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-flzkd\" (UID: \"3a087c70-a94f-45ee-8469-e19cac8100a4\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-flzkd"
Jul 30 18:20:30.586567 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:20:30.586337    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a087c70-a94f-45ee-8469-e19cac8100a4-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-flzkd\" (UID: \"3a087c70-a94f-45ee-8469-e19cac8100a4\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-flzkd"
Jul 30 18:20:30.586567 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:20:30.586371    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a087c70-a94f-45ee-8469-e19cac8100a4-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-flzkd\" (UID: \"3a087c70-a94f-45ee-8469-e19cac8100a4\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-flzkd"
Jul 30 18:20:30.586567 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:20:30.586414    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a087c70-a94f-45ee-8469-e19cac8100a4-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-flzkd\" (UID: \"3a087c70-a94f-45ee-8469-e19cac8100a4\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-flzkd"
Jul 30 18:20:30.586567 ip-10-0-143-190 kubenswrapper[2555]: E0730 18:20:30.586521    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-serving-cert: secret "xgboost-v2-mlserver-predictor-serving-cert" not found
Jul 30 18:20:30.586835 ip-10-0-143-190 kubenswrapper[2555]: E0730 18:20:30.586589    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a087c70-a94f-45ee-8469-e19cac8100a4-proxy-tls podName:3a087c70-a94f-45ee-8469-e19cac8100a4 nodeName:}" failed. No retries permitted until 2026-07-30 18:20:31.086569012 +0000 UTC m=+4805.573936408 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3a087c70-a94f-45ee-8469-e19cac8100a4-proxy-tls") pod "xgboost-v2-mlserver-predictor-59944bcf4c-flzkd" (UID: "3a087c70-a94f-45ee-8469-e19cac8100a4") : secret "xgboost-v2-mlserver-predictor-serving-cert" not found
Jul 30 18:20:30.586835 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:20:30.586739    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a087c70-a94f-45ee-8469-e19cac8100a4-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-flzkd\" (UID: \"3a087c70-a94f-45ee-8469-e19cac8100a4\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-flzkd"
Jul 30 18:20:30.587074 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:20:30.587050    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a087c70-a94f-45ee-8469-e19cac8100a4-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-flzkd\" (UID: \"3a087c70-a94f-45ee-8469-e19cac8100a4\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-flzkd"
Jul 30 18:20:30.587170 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:20:30.587146    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a087c70-a94f-45ee-8469-e19cac8100a4-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-flzkd\" (UID: \"3a087c70-a94f-45ee-8469-e19cac8100a4\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-flzkd"
Jul 30 18:20:30.597460 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:20:30.597438    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbq96\" (UniqueName: \"kubernetes.io/projected/3a087c70-a94f-45ee-8469-e19cac8100a4-kube-api-access-tbq96\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-flzkd\" (UID: \"3a087c70-a94f-45ee-8469-e19cac8100a4\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-flzkd"
Jul 30 18:20:30.732138 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:20:30.732038    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:20:31.091082 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:20:31.091051    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a087c70-a94f-45ee-8469-e19cac8100a4-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-flzkd\" (UID: \"3a087c70-a94f-45ee-8469-e19cac8100a4\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-flzkd"
Jul 30 18:20:31.093390 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:20:31.093367    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a087c70-a94f-45ee-8469-e19cac8100a4-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-flzkd\" (UID: \"3a087c70-a94f-45ee-8469-e19cac8100a4\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-flzkd"
Jul 30 18:20:31.320112 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:20:31.320062    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-flzkd"
Jul 30 18:20:31.460810 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:20:31.460780    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-flzkd"]
Jul 30 18:20:31.462925 ip-10-0-143-190 kubenswrapper[2555]: W0730 18:20:31.462900    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a087c70_a94f_45ee_8469_e19cac8100a4.slice/crio-1d1466bd86dc62650ab950edefb5d84329467da255247aab535e2d7a195d561c WatchSource:0}: Error finding container 1d1466bd86dc62650ab950edefb5d84329467da255247aab535e2d7a195d561c: Status 404 returned error can't find the container with id 1d1466bd86dc62650ab950edefb5d84329467da255247aab535e2d7a195d561c
Jul 30 18:20:31.781792 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:20:31.781766    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-shb8f"
Jul 30 18:20:31.899060 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:20:31.898947    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pdrlt\" (UniqueName: \"kubernetes.io/projected/f59dedb6-adb5-409a-bbc6-c487b9426a0b-kube-api-access-pdrlt\") pod \"f59dedb6-adb5-409a-bbc6-c487b9426a0b\" (UID: \"f59dedb6-adb5-409a-bbc6-c487b9426a0b\") "
Jul 30 18:20:31.899060 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:20:31.899053    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f59dedb6-adb5-409a-bbc6-c487b9426a0b-cabundle-cert\") pod \"f59dedb6-adb5-409a-bbc6-c487b9426a0b\" (UID: \"f59dedb6-adb5-409a-bbc6-c487b9426a0b\") "
Jul 30 18:20:31.899287 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:20:31.899112    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f59dedb6-adb5-409a-bbc6-c487b9426a0b-kserve-provision-location\") pod \"f59dedb6-adb5-409a-bbc6-c487b9426a0b\" (UID: \"f59dedb6-adb5-409a-bbc6-c487b9426a0b\") "
Jul 30 18:20:31.899287 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:20:31.899153    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f59dedb6-adb5-409a-bbc6-c487b9426a0b-kube-rbac-proxy-sar-config\") pod \"f59dedb6-adb5-409a-bbc6-c487b9426a0b\" (UID: \"f59dedb6-adb5-409a-bbc6-c487b9426a0b\") "
Jul 30 18:20:31.899287 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:20:31.899173    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f59dedb6-adb5-409a-bbc6-c487b9426a0b-proxy-tls\") pod \"f59dedb6-adb5-409a-bbc6-c487b9426a0b\" (UID: \"f59dedb6-adb5-409a-bbc6-c487b9426a0b\") "
Jul 30 18:20:31.899532 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:20:31.899498    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f59dedb6-adb5-409a-bbc6-c487b9426a0b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f59dedb6-adb5-409a-bbc6-c487b9426a0b" (UID: "f59dedb6-adb5-409a-bbc6-c487b9426a0b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 18:20:31.899584 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:20:31.899547    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f59dedb6-adb5-409a-bbc6-c487b9426a0b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f59dedb6-adb5-409a-bbc6-c487b9426a0b" (UID: "f59dedb6-adb5-409a-bbc6-c487b9426a0b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 18:20:31.899651 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:20:31.899626    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f59dedb6-adb5-409a-bbc6-c487b9426a0b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f59dedb6-adb5-409a-bbc6-c487b9426a0b" (UID: "f59dedb6-adb5-409a-bbc6-c487b9426a0b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 18:20:31.901137 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:20:31.901115    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f59dedb6-adb5-409a-bbc6-c487b9426a0b-kube-api-access-pdrlt" (OuterVolumeSpecName: "kube-api-access-pdrlt") pod "f59dedb6-adb5-409a-bbc6-c487b9426a0b" (UID: "f59dedb6-adb5-409a-bbc6-c487b9426a0b"). InnerVolumeSpecName "kube-api-access-pdrlt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 18:20:31.901363 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:20:31.901340    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f59dedb6-adb5-409a-bbc6-c487b9426a0b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f59dedb6-adb5-409a-bbc6-c487b9426a0b" (UID: "f59dedb6-adb5-409a-bbc6-c487b9426a0b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 18:20:32.000641 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:20:32.000609    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f59dedb6-adb5-409a-bbc6-c487b9426a0b-kserve-provision-location\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 18:20:32.000641 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:20:32.000637    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f59dedb6-adb5-409a-bbc6-c487b9426a0b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 18:20:32.000641 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:20:32.000646    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f59dedb6-adb5-409a-bbc6-c487b9426a0b-proxy-tls\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 18:20:32.000871 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:20:32.000654    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdrlt\" (UniqueName: \"kubernetes.io/projected/f59dedb6-adb5-409a-bbc6-c487b9426a0b-kube-api-access-pdrlt\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 18:20:32.000871 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:20:32.000664    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f59dedb6-adb5-409a-bbc6-c487b9426a0b-cabundle-cert\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 18:20:32.468928 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:20:32.468892    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-flzkd" event={"ID":"3a087c70-a94f-45ee-8469-e19cac8100a4","Type":"ContainerStarted","Data":"bbe08278206163168b3eec87a90a444742b0862b5340201ead292a6ae6e54476"}
Jul 30 18:20:32.469427 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:20:32.468950    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-flzkd" event={"ID":"3a087c70-a94f-45ee-8469-e19cac8100a4","Type":"ContainerStarted","Data":"1d1466bd86dc62650ab950edefb5d84329467da255247aab535e2d7a195d561c"}
Jul 30 18:20:32.470757 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:20:32.470731    2555 generic.go:356] "Generic (PLEG): container finished" podID="f59dedb6-adb5-409a-bbc6-c487b9426a0b" containerID="18f990bded7efa9254cbb95d6ad55087748a00282165330189accefb3ca2561f" exitCode=0
Jul 30 18:20:32.470891 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:20:32.470777    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-shb8f"
Jul 30 18:20:32.470891 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:20:32.470799    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-shb8f" event={"ID":"f59dedb6-adb5-409a-bbc6-c487b9426a0b","Type":"ContainerDied","Data":"18f990bded7efa9254cbb95d6ad55087748a00282165330189accefb3ca2561f"}
Jul 30 18:20:32.470891 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:20:32.470858    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-shb8f" event={"ID":"f59dedb6-adb5-409a-bbc6-c487b9426a0b","Type":"ContainerDied","Data":"0ef46c6af47c25d1aaf4ff688198a133989f9ae0a5d1e03c15df7451510dbbf0"}
Jul 30 18:20:32.471067 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:20:32.470896    2555 scope.go:117] "RemoveContainer" containerID="d82dbdad04287591b623f851f4218ee52fc2f6038dd58cc1f47d5ecbee0ea7c0"
Jul 30 18:20:32.481442 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:20:32.481424    2555 scope.go:117] "RemoveContainer" containerID="18f990bded7efa9254cbb95d6ad55087748a00282165330189accefb3ca2561f"
Jul 30 18:20:32.489179 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:20:32.489161    2555 scope.go:117] "RemoveContainer" containerID="2796cc533f348744b3d1fb26293feb2818d747981a75e3c644f5efc361d04acb"
Jul 30 18:20:32.497108 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:20:32.497093    2555 scope.go:117] "RemoveContainer" containerID="d82dbdad04287591b623f851f4218ee52fc2f6038dd58cc1f47d5ecbee0ea7c0"
Jul 30 18:20:32.497379 ip-10-0-143-190 kubenswrapper[2555]: E0730 18:20:32.497349    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d82dbdad04287591b623f851f4218ee52fc2f6038dd58cc1f47d5ecbee0ea7c0\": container with ID starting with d82dbdad04287591b623f851f4218ee52fc2f6038dd58cc1f47d5ecbee0ea7c0 not found: ID does not exist" containerID="d82dbdad04287591b623f851f4218ee52fc2f6038dd58cc1f47d5ecbee0ea7c0"
Jul 30 18:20:32.497452 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:20:32.497382    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d82dbdad04287591b623f851f4218ee52fc2f6038dd58cc1f47d5ecbee0ea7c0"} err="failed to get container status \"d82dbdad04287591b623f851f4218ee52fc2f6038dd58cc1f47d5ecbee0ea7c0\": rpc error: code = NotFound desc = could not find container \"d82dbdad04287591b623f851f4218ee52fc2f6038dd58cc1f47d5ecbee0ea7c0\": container with ID starting with d82dbdad04287591b623f851f4218ee52fc2f6038dd58cc1f47d5ecbee0ea7c0 not found: ID does not exist"
Jul 30 18:20:32.497452 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:20:32.497401    2555 scope.go:117] "RemoveContainer" containerID="18f990bded7efa9254cbb95d6ad55087748a00282165330189accefb3ca2561f"
Jul 30 18:20:32.497683 ip-10-0-143-190 kubenswrapper[2555]: E0730 18:20:32.497665    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18f990bded7efa9254cbb95d6ad55087748a00282165330189accefb3ca2561f\": container with ID starting with 18f990bded7efa9254cbb95d6ad55087748a00282165330189accefb3ca2561f not found: ID does not exist" containerID="18f990bded7efa9254cbb95d6ad55087748a00282165330189accefb3ca2561f"
Jul 30 18:20:32.497775 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:20:32.497695    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18f990bded7efa9254cbb95d6ad55087748a00282165330189accefb3ca2561f"} err="failed to get container status \"18f990bded7efa9254cbb95d6ad55087748a00282165330189accefb3ca2561f\": rpc error: code = NotFound desc = could not find container \"18f990bded7efa9254cbb95d6ad55087748a00282165330189accefb3ca2561f\": container with ID starting with 18f990bded7efa9254cbb95d6ad55087748a00282165330189accefb3ca2561f not found: ID does not exist"
Jul 30 18:20:32.497775 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:20:32.497719    2555 scope.go:117] "RemoveContainer" containerID="2796cc533f348744b3d1fb26293feb2818d747981a75e3c644f5efc361d04acb"
Jul 30 18:20:32.498090 ip-10-0-143-190 kubenswrapper[2555]: E0730 18:20:32.498050    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2796cc533f348744b3d1fb26293feb2818d747981a75e3c644f5efc361d04acb\": container with ID starting with 2796cc533f348744b3d1fb26293feb2818d747981a75e3c644f5efc361d04acb not found: ID does not exist" containerID="2796cc533f348744b3d1fb26293feb2818d747981a75e3c644f5efc361d04acb"
Jul 30 18:20:32.498161 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:20:32.498103    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2796cc533f348744b3d1fb26293feb2818d747981a75e3c644f5efc361d04acb"} err="failed to get container status \"2796cc533f348744b3d1fb26293feb2818d747981a75e3c644f5efc361d04acb\": rpc error: code = NotFound desc = could not find container \"2796cc533f348744b3d1fb26293feb2818d747981a75e3c644f5efc361d04acb\": container with ID starting with 2796cc533f348744b3d1fb26293feb2818d747981a75e3c644f5efc361d04acb not found: ID does not exist"
Jul 30 18:20:32.505909 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:20:32.505889    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-shb8f"]
Jul 30 18:20:32.513991 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:20:32.513965    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-shb8f"]
Jul 30 18:20:34.061609 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:20:34.061582    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f59dedb6-adb5-409a-bbc6-c487b9426a0b" path="/var/lib/kubelet/pods/f59dedb6-adb5-409a-bbc6-c487b9426a0b/volumes"
Jul 30 18:20:35.486153 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:20:35.486121    2555 generic.go:356] "Generic (PLEG): container finished" podID="3a087c70-a94f-45ee-8469-e19cac8100a4" containerID="bbe08278206163168b3eec87a90a444742b0862b5340201ead292a6ae6e54476" exitCode=0
Jul 30 18:20:35.486535 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:20:35.486190    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-flzkd" event={"ID":"3a087c70-a94f-45ee-8469-e19cac8100a4","Type":"ContainerDied","Data":"bbe08278206163168b3eec87a90a444742b0862b5340201ead292a6ae6e54476"}
Jul 30 18:20:36.492936 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:20:36.492904    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-flzkd" event={"ID":"3a087c70-a94f-45ee-8469-e19cac8100a4","Type":"ContainerStarted","Data":"a77f2055bd42260c726b4dede25ce43e7e1b21a520db654d1ebd79ebdd238d88"}
Jul 30 18:20:36.493349 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:20:36.492968    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-flzkd" event={"ID":"3a087c70-a94f-45ee-8469-e19cac8100a4","Type":"ContainerStarted","Data":"eea5de24a68549755de287b33dbe2bde9f25ab068eb13b4aa5c29c539ce9c1b9"}
Jul 30 18:20:36.493349 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:20:36.493189    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-flzkd"
Jul 30 18:20:36.493349 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:20:36.493240    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-flzkd"
Jul 30 18:20:36.516446 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:20:36.516401    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-flzkd" podStartSLOduration=2.691757382 podStartE2EDuration="6.516384833s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.824627451s" isStatefulPod=false podCreationTimestamp="2026-07-30 18:20:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 18:20:36.515179375 +0000 UTC m=+4811.002546777" watchObservedRunningTime="2026-07-30 18:20:36.516384833 +0000 UTC m=+4811.003752235"
Jul 30 18:20:42.503111 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:20:42.503081    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-flzkd"
Jul 30 18:21:12.508214 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:21:12.508184    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-flzkd"
Jul 30 18:21:20.518311 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:21:20.518227    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-flzkd"]
Jul 30 18:21:20.518828 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:21:20.518483    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-flzkd" podUID="3a087c70-a94f-45ee-8469-e19cac8100a4" containerName="kserve-container" containerID="cri-o://eea5de24a68549755de287b33dbe2bde9f25ab068eb13b4aa5c29c539ce9c1b9" gracePeriod=30
Jul 30 18:21:20.518828 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:21:20.518549    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-flzkd" podUID="3a087c70-a94f-45ee-8469-e19cac8100a4" containerName="kube-rbac-proxy" containerID="cri-o://a77f2055bd42260c726b4dede25ce43e7e1b21a520db654d1ebd79ebdd238d88" gracePeriod=30
Jul 30 18:21:20.534227 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:21:20.534204    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:21:20.699551 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:21:20.699516    2555 generic.go:356] "Generic (PLEG): container finished" podID="3a087c70-a94f-45ee-8469-e19cac8100a4" containerID="a77f2055bd42260c726b4dede25ce43e7e1b21a520db654d1ebd79ebdd238d88" exitCode=2
Jul 30 18:21:20.699715 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:21:20.699589    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-flzkd" event={"ID":"3a087c70-a94f-45ee-8469-e19cac8100a4","Type":"ContainerDied","Data":"a77f2055bd42260c726b4dede25ce43e7e1b21a520db654d1ebd79ebdd238d88"}
Jul 30 18:21:22.497770 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:21:22.497732    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-flzkd" podUID="3a087c70-a94f-45ee-8469-e19cac8100a4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.52:8643/healthz\": dial tcp 10.133.0.52:8643: connect: connection refused"
Jul 30 18:21:25.615318 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:21:25.615282    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:21:25.961260 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:21:25.961182    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:21:27.064966 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:21:27.064944    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-flzkd"
Jul 30 18:21:27.162212 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:21:27.162121    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a087c70-a94f-45ee-8469-e19cac8100a4-cabundle-cert\") pod \"3a087c70-a94f-45ee-8469-e19cac8100a4\" (UID: \"3a087c70-a94f-45ee-8469-e19cac8100a4\") "
Jul 30 18:21:27.162212 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:21:27.162202    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a087c70-a94f-45ee-8469-e19cac8100a4-kserve-provision-location\") pod \"3a087c70-a94f-45ee-8469-e19cac8100a4\" (UID: \"3a087c70-a94f-45ee-8469-e19cac8100a4\") "
Jul 30 18:21:27.162455 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:21:27.162229    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a087c70-a94f-45ee-8469-e19cac8100a4-proxy-tls\") pod \"3a087c70-a94f-45ee-8469-e19cac8100a4\" (UID: \"3a087c70-a94f-45ee-8469-e19cac8100a4\") "
Jul 30 18:21:27.162455 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:21:27.162275    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a087c70-a94f-45ee-8469-e19cac8100a4-kube-rbac-proxy-sar-config\") pod \"3a087c70-a94f-45ee-8469-e19cac8100a4\" (UID: \"3a087c70-a94f-45ee-8469-e19cac8100a4\") "
Jul 30 18:21:27.162455 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:21:27.162302    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tbq96\" (UniqueName: \"kubernetes.io/projected/3a087c70-a94f-45ee-8469-e19cac8100a4-kube-api-access-tbq96\") pod \"3a087c70-a94f-45ee-8469-e19cac8100a4\" (UID: \"3a087c70-a94f-45ee-8469-e19cac8100a4\") "
Jul 30 18:21:27.162616 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:21:27.162562    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a087c70-a94f-45ee-8469-e19cac8100a4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3a087c70-a94f-45ee-8469-e19cac8100a4" (UID: "3a087c70-a94f-45ee-8469-e19cac8100a4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 18:21:27.162616 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:21:27.162568    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a087c70-a94f-45ee-8469-e19cac8100a4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3a087c70-a94f-45ee-8469-e19cac8100a4" (UID: "3a087c70-a94f-45ee-8469-e19cac8100a4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 18:21:27.162759 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:21:27.162732    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a087c70-a94f-45ee-8469-e19cac8100a4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3a087c70-a94f-45ee-8469-e19cac8100a4" (UID: "3a087c70-a94f-45ee-8469-e19cac8100a4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 18:21:27.164420 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:21:27.164397    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a087c70-a94f-45ee-8469-e19cac8100a4-kube-api-access-tbq96" (OuterVolumeSpecName: "kube-api-access-tbq96") pod "3a087c70-a94f-45ee-8469-e19cac8100a4" (UID: "3a087c70-a94f-45ee-8469-e19cac8100a4"). InnerVolumeSpecName "kube-api-access-tbq96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 18:21:27.164505 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:21:27.164475    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a087c70-a94f-45ee-8469-e19cac8100a4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3a087c70-a94f-45ee-8469-e19cac8100a4" (UID: "3a087c70-a94f-45ee-8469-e19cac8100a4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 18:21:27.262963 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:21:27.262930    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a087c70-a94f-45ee-8469-e19cac8100a4-kserve-provision-location\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 18:21:27.262963 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:21:27.262956    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a087c70-a94f-45ee-8469-e19cac8100a4-proxy-tls\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 18:21:27.262963 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:21:27.262967    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a087c70-a94f-45ee-8469-e19cac8100a4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 18:21:27.262963 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:21:27.262975    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbq96\" (UniqueName: \"kubernetes.io/projected/3a087c70-a94f-45ee-8469-e19cac8100a4-kube-api-access-tbq96\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 18:21:27.263311 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:21:27.262984    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a087c70-a94f-45ee-8469-e19cac8100a4-cabundle-cert\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 18:21:27.738191 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:21:27.738157    2555 generic.go:356] "Generic (PLEG): container finished" podID="3a087c70-a94f-45ee-8469-e19cac8100a4" containerID="eea5de24a68549755de287b33dbe2bde9f25ab068eb13b4aa5c29c539ce9c1b9" exitCode=0
Jul 30 18:21:27.738368 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:21:27.738214    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-flzkd" event={"ID":"3a087c70-a94f-45ee-8469-e19cac8100a4","Type":"ContainerDied","Data":"eea5de24a68549755de287b33dbe2bde9f25ab068eb13b4aa5c29c539ce9c1b9"}
Jul 30 18:21:27.738368 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:21:27.738215    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-flzkd"
Jul 30 18:21:27.738368 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:21:27.738260    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-flzkd" event={"ID":"3a087c70-a94f-45ee-8469-e19cac8100a4","Type":"ContainerDied","Data":"1d1466bd86dc62650ab950edefb5d84329467da255247aab535e2d7a195d561c"}
Jul 30 18:21:27.738368 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:21:27.738285    2555 scope.go:117] "RemoveContainer" containerID="a77f2055bd42260c726b4dede25ce43e7e1b21a520db654d1ebd79ebdd238d88"
Jul 30 18:21:27.748401 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:21:27.748381    2555 scope.go:117] "RemoveContainer" containerID="eea5de24a68549755de287b33dbe2bde9f25ab068eb13b4aa5c29c539ce9c1b9"
Jul 30 18:21:27.756332 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:21:27.756314    2555 scope.go:117] "RemoveContainer" containerID="bbe08278206163168b3eec87a90a444742b0862b5340201ead292a6ae6e54476"
Jul 30 18:21:27.764640 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:21:27.764613    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-flzkd"]
Jul 30 18:21:27.765356 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:21:27.765333    2555 scope.go:117] "RemoveContainer" containerID="a77f2055bd42260c726b4dede25ce43e7e1b21a520db654d1ebd79ebdd238d88"
Jul 30 18:21:27.765679 ip-10-0-143-190 kubenswrapper[2555]: E0730 18:21:27.765661    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a77f2055bd42260c726b4dede25ce43e7e1b21a520db654d1ebd79ebdd238d88\": container with ID starting with a77f2055bd42260c726b4dede25ce43e7e1b21a520db654d1ebd79ebdd238d88 not found: ID does not exist" containerID="a77f2055bd42260c726b4dede25ce43e7e1b21a520db654d1ebd79ebdd238d88"
Jul 30 18:21:27.765737 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:21:27.765689    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a77f2055bd42260c726b4dede25ce43e7e1b21a520db654d1ebd79ebdd238d88"} err="failed to get container status \"a77f2055bd42260c726b4dede25ce43e7e1b21a520db654d1ebd79ebdd238d88\": rpc error: code = NotFound desc = could not find container \"a77f2055bd42260c726b4dede25ce43e7e1b21a520db654d1ebd79ebdd238d88\": container with ID starting with a77f2055bd42260c726b4dede25ce43e7e1b21a520db654d1ebd79ebdd238d88 not found: ID does not exist"
Jul 30 18:21:27.765737 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:21:27.765710    2555 scope.go:117] "RemoveContainer" containerID="eea5de24a68549755de287b33dbe2bde9f25ab068eb13b4aa5c29c539ce9c1b9"
Jul 30 18:21:27.766151 ip-10-0-143-190 kubenswrapper[2555]: E0730 18:21:27.766032    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eea5de24a68549755de287b33dbe2bde9f25ab068eb13b4aa5c29c539ce9c1b9\": container with ID starting with eea5de24a68549755de287b33dbe2bde9f25ab068eb13b4aa5c29c539ce9c1b9 not found: ID does not exist" containerID="eea5de24a68549755de287b33dbe2bde9f25ab068eb13b4aa5c29c539ce9c1b9"
Jul 30 18:21:27.766151 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:21:27.766064    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eea5de24a68549755de287b33dbe2bde9f25ab068eb13b4aa5c29c539ce9c1b9"} err="failed to get container status \"eea5de24a68549755de287b33dbe2bde9f25ab068eb13b4aa5c29c539ce9c1b9\": rpc error: code = NotFound desc = could not find container \"eea5de24a68549755de287b33dbe2bde9f25ab068eb13b4aa5c29c539ce9c1b9\": container with ID starting with eea5de24a68549755de287b33dbe2bde9f25ab068eb13b4aa5c29c539ce9c1b9 not found: ID does not exist"
Jul 30 18:21:27.766151 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:21:27.766085    2555 scope.go:117] "RemoveContainer" containerID="bbe08278206163168b3eec87a90a444742b0862b5340201ead292a6ae6e54476"
Jul 30 18:21:27.766364 ip-10-0-143-190 kubenswrapper[2555]: E0730 18:21:27.766347    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbe08278206163168b3eec87a90a444742b0862b5340201ead292a6ae6e54476\": container with ID starting with bbe08278206163168b3eec87a90a444742b0862b5340201ead292a6ae6e54476 not found: ID does not exist" containerID="bbe08278206163168b3eec87a90a444742b0862b5340201ead292a6ae6e54476"
Jul 30 18:21:27.766405 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:21:27.766370    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbe08278206163168b3eec87a90a444742b0862b5340201ead292a6ae6e54476"} err="failed to get container status \"bbe08278206163168b3eec87a90a444742b0862b5340201ead292a6ae6e54476\": rpc error: code = NotFound desc = could not find container \"bbe08278206163168b3eec87a90a444742b0862b5340201ead292a6ae6e54476\": container with ID starting with bbe08278206163168b3eec87a90a444742b0862b5340201ead292a6ae6e54476 not found: ID does not exist"
Jul 30 18:21:27.766965 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:21:27.766949    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-flzkd"]
Jul 30 18:21:28.060796 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:21:28.060767    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a087c70-a94f-45ee-8469-e19cac8100a4" path="/var/lib/kubelet/pods/3a087c70-a94f-45ee-8469-e19cac8100a4/volumes"
Jul 30 18:22:45.760322 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:22:45.760290    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:22:50.879305 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:22:50.879267    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:22:50.911888 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:22:50.911853    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ghgwx"]
Jul 30 18:22:50.912312 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:22:50.912297    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f59dedb6-adb5-409a-bbc6-c487b9426a0b" containerName="kserve-container"
Jul 30 18:22:50.912409 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:22:50.912313    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f59dedb6-adb5-409a-bbc6-c487b9426a0b" containerName="kserve-container"
Jul 30 18:22:50.912409 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:22:50.912357    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f59dedb6-adb5-409a-bbc6-c487b9426a0b" containerName="storage-initializer"
Jul 30 18:22:50.912409 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:22:50.912363    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f59dedb6-adb5-409a-bbc6-c487b9426a0b" containerName="storage-initializer"
Jul 30 18:22:50.912409 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:22:50.912369    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a087c70-a94f-45ee-8469-e19cac8100a4" containerName="kserve-container"
Jul 30 18:22:50.912409 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:22:50.912377    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a087c70-a94f-45ee-8469-e19cac8100a4" containerName="kserve-container"
Jul 30 18:22:50.912409 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:22:50.912389    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a087c70-a94f-45ee-8469-e19cac8100a4" containerName="storage-initializer"
Jul 30 18:22:50.912409 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:22:50.912395    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a087c70-a94f-45ee-8469-e19cac8100a4" containerName="storage-initializer"
Jul 30 18:22:50.912409 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:22:50.912409    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f59dedb6-adb5-409a-bbc6-c487b9426a0b" containerName="kube-rbac-proxy"
Jul 30 18:22:50.912715 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:22:50.912414    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f59dedb6-adb5-409a-bbc6-c487b9426a0b" containerName="kube-rbac-proxy"
Jul 30 18:22:50.912715 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:22:50.912420    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a087c70-a94f-45ee-8469-e19cac8100a4" containerName="kube-rbac-proxy"
Jul 30 18:22:50.912715 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:22:50.912425    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a087c70-a94f-45ee-8469-e19cac8100a4" containerName="kube-rbac-proxy"
Jul 30 18:22:50.912715 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:22:50.912495    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a087c70-a94f-45ee-8469-e19cac8100a4" containerName="kube-rbac-proxy"
Jul 30 18:22:50.912715 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:22:50.912508    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="f59dedb6-adb5-409a-bbc6-c487b9426a0b" containerName="kserve-container"
Jul 30 18:22:50.912715 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:22:50.912514    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a087c70-a94f-45ee-8469-e19cac8100a4" containerName="kserve-container"
Jul 30 18:22:50.912715 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:22:50.912522    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="f59dedb6-adb5-409a-bbc6-c487b9426a0b" containerName="kube-rbac-proxy"
Jul 30 18:22:50.915947 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:22:50.915930    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ghgwx"
Jul 30 18:22:50.918655 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:22:50.918625    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 30 18:22:50.918772 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:22:50.918682    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 30 18:22:50.918772 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:22:50.918625    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\""
Jul 30 18:22:50.918772 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:22:50.918741    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-plw5t\""
Jul 30 18:22:50.919855 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:22:50.919840    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 30 18:22:50.919942 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:22:50.919875    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 30 18:22:50.919942 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:22:50.919923    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-predictor-serving-cert\""
Jul 30 18:22:50.928000 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:22:50.927980    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ghgwx"]
Jul 30 18:22:50.978658 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:22:50.978626    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:22:50.981741 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:22:50.981718    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-ghgwx\" (UID: \"8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ghgwx"
Jul 30 18:22:50.981868 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:22:50.981771    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-ghgwx\" (UID: \"8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ghgwx"
Jul 30 18:22:50.981914 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:22:50.981878    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-ghgwx\" (UID: \"8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ghgwx"
Jul 30 18:22:50.981950 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:22:50.981928    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqwr9\" (UniqueName: \"kubernetes.io/projected/8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e-kube-api-access-qqwr9\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-ghgwx\" (UID: \"8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ghgwx"
Jul 30 18:22:50.981985 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:22:50.981948    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-ghgwx\" (UID: \"8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ghgwx"
Jul 30 18:22:51.082555 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:22:51.082516    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-ghgwx\" (UID: \"8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ghgwx"
Jul 30 18:22:51.082753 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:22:51.082570    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqwr9\" (UniqueName: \"kubernetes.io/projected/8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e-kube-api-access-qqwr9\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-ghgwx\" (UID: \"8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ghgwx"
Jul 30 18:22:51.082753 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:22:51.082597    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-ghgwx\" (UID: \"8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ghgwx"
Jul 30 18:22:51.082753 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:22:51.082621    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-ghgwx\" (UID: \"8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ghgwx"
Jul 30 18:22:51.082934 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:22:51.082771    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-ghgwx\" (UID: \"8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ghgwx"
Jul 30 18:22:51.083064 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:22:51.083045    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-ghgwx\" (UID: \"8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ghgwx"
Jul 30 18:22:51.083333 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:22:51.083312    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-ghgwx\" (UID: \"8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ghgwx"
Jul 30 18:22:51.083445 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:22:51.083404    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-ghgwx\" (UID: \"8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ghgwx"
Jul 30 18:22:51.085203 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:22:51.085188    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-ghgwx\" (UID: \"8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ghgwx"
Jul 30 18:22:51.093061 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:22:51.093036    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqwr9\" (UniqueName: \"kubernetes.io/projected/8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e-kube-api-access-qqwr9\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-ghgwx\" (UID: \"8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ghgwx"
Jul 30 18:22:51.228365 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:22:51.228272    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ghgwx"
Jul 30 18:22:51.367721 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:22:51.367692    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ghgwx"]
Jul 30 18:22:51.369387 ip-10-0-143-190 kubenswrapper[2555]: W0730 18:22:51.369359    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b1c74a5_6fb0_40a1_bf0d_e41e96c7746e.slice/crio-0fa7b4fe519b53b710ebf608edbd32dd3973729b7b0366833d07e279b7b271cc WatchSource:0}: Error finding container 0fa7b4fe519b53b710ebf608edbd32dd3973729b7b0366833d07e279b7b271cc: Status 404 returned error can't find the container with id 0fa7b4fe519b53b710ebf608edbd32dd3973729b7b0366833d07e279b7b271cc
Jul 30 18:22:52.119786 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:22:52.119743    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ghgwx" event={"ID":"8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e","Type":"ContainerStarted","Data":"0cd766236d6ca178706afacda90a2553519ebec627289b515772aea518c9e30a"}
Jul 30 18:22:52.120198 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:22:52.119819    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ghgwx" event={"ID":"8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e","Type":"ContainerStarted","Data":"0fa7b4fe519b53b710ebf608edbd32dd3973729b7b0366833d07e279b7b271cc"}
Jul 30 18:22:56.139543 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:22:56.139510    2555 generic.go:356] "Generic (PLEG): container finished" podID="8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e" containerID="0cd766236d6ca178706afacda90a2553519ebec627289b515772aea518c9e30a" exitCode=0
Jul 30 18:22:56.140060 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:22:56.139562    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ghgwx" event={"ID":"8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e","Type":"ContainerDied","Data":"0cd766236d6ca178706afacda90a2553519ebec627289b515772aea518c9e30a"}
Jul 30 18:22:57.145572 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:22:57.145542    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ghgwx" event={"ID":"8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e","Type":"ContainerStarted","Data":"baf16852f97356370c0536681bc49083129a6ab8cbed3128915e956ace545c50"}
Jul 30 18:22:57.145959 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:22:57.145599    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ghgwx" event={"ID":"8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e","Type":"ContainerStarted","Data":"9e89f4e4612306f775edf27cdf9032854ceb6b1163894c85a826eba7b778dca3"}
Jul 30 18:22:57.145959 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:22:57.145785    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ghgwx"
Jul 30 18:22:57.170092 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:22:57.170040    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ghgwx" podStartSLOduration=3.362330533 podStartE2EDuration="7.16999764s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.807667107s" isStatefulPod=false podCreationTimestamp="2026-07-30 18:22:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 18:22:57.168711956 +0000 UTC m=+4951.656079373" watchObservedRunningTime="2026-07-30 18:22:57.16999764 +0000 UTC m=+4951.657365040"
Jul 30 18:22:58.150399 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:22:58.150369    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ghgwx"
Jul 30 18:23:04.160275 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:23:04.160242    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ghgwx"
Jul 30 18:23:34.203288 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:23:34.203243    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ghgwx" podUID="8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Jul 30 18:23:44.163301 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:23:44.163268    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ghgwx"
Jul 30 18:23:51.034903 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:23:51.034869    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ghgwx"]
Jul 30 18:23:51.035354 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:23:51.035128    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ghgwx" podUID="8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e" containerName="kserve-container" containerID="cri-o://9e89f4e4612306f775edf27cdf9032854ceb6b1163894c85a826eba7b778dca3" gracePeriod=30
Jul 30 18:23:51.035354 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:23:51.035152    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ghgwx" podUID="8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e" containerName="kube-rbac-proxy" containerID="cri-o://baf16852f97356370c0536681bc49083129a6ab8cbed3128915e956ace545c50" gracePeriod=30
Jul 30 18:23:51.061405 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:23:51.061377    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:23:51.393388 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:23:51.393358    2555 generic.go:356] "Generic (PLEG): container finished" podID="8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e" containerID="baf16852f97356370c0536681bc49083129a6ab8cbed3128915e956ace545c50" exitCode=2
Jul 30 18:23:51.393579 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:23:51.393434    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ghgwx" event={"ID":"8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e","Type":"ContainerDied","Data":"baf16852f97356370c0536681bc49083129a6ab8cbed3128915e956ace545c50"}
Jul 30 18:23:54.155465 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:23:54.155418    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ghgwx" podUID="8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.53:8643/healthz\": dial tcp 10.133.0.53:8643: connect: connection refused"
Jul 30 18:23:56.168436 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:23:56.168400    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:23:56.241543 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:23:56.241509    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:23:58.432071 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:23:58.432037    2555 generic.go:356] "Generic (PLEG): container finished" podID="8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e" containerID="9e89f4e4612306f775edf27cdf9032854ceb6b1163894c85a826eba7b778dca3" exitCode=0
Jul 30 18:23:58.432429 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:23:58.432103    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ghgwx" event={"ID":"8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e","Type":"ContainerDied","Data":"9e89f4e4612306f775edf27cdf9032854ceb6b1163894c85a826eba7b778dca3"}
Jul 30 18:23:58.485851 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:23:58.485828    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ghgwx"
Jul 30 18:23:58.590041 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:23:58.589931    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e-kserve-provision-location\") pod \"8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e\" (UID: \"8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e\") "
Jul 30 18:23:58.590041 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:23:58.589983    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qqwr9\" (UniqueName: \"kubernetes.io/projected/8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e-kube-api-access-qqwr9\") pod \"8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e\" (UID: \"8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e\") "
Jul 30 18:23:58.590254 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:23:58.590087    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e-kube-rbac-proxy-sar-config\") pod \"8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e\" (UID: \"8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e\") "
Jul 30 18:23:58.590254 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:23:58.590130    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e-cabundle-cert\") pod \"8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e\" (UID: \"8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e\") "
Jul 30 18:23:58.590254 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:23:58.590173    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e-proxy-tls\") pod \"8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e\" (UID: \"8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e\") "
Jul 30 18:23:58.590408 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:23:58.590310    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e" (UID: "8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 18:23:58.590504 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:23:58.590406    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e" (UID: "8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 18:23:58.590504 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:23:58.590456    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e" (UID: "8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 18:23:58.592170 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:23:58.592146    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e-kube-api-access-qqwr9" (OuterVolumeSpecName: "kube-api-access-qqwr9") pod "8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e" (UID: "8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e"). InnerVolumeSpecName "kube-api-access-qqwr9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 18:23:58.592278 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:23:58.592146    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e" (UID: "8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 18:23:58.691781 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:23:58.691738    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e-kserve-provision-location\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 18:23:58.691781 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:23:58.691778    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqwr9\" (UniqueName: \"kubernetes.io/projected/8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e-kube-api-access-qqwr9\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 18:23:58.691781 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:23:58.691789    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 18:23:58.691781 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:23:58.691799    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e-cabundle-cert\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 18:23:58.692074 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:23:58.691808    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e-proxy-tls\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Jul 30 18:23:59.438516 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:23:59.438491    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ghgwx"
Jul 30 18:23:59.438953 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:23:59.438489    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ghgwx" event={"ID":"8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e","Type":"ContainerDied","Data":"0fa7b4fe519b53b710ebf608edbd32dd3973729b7b0366833d07e279b7b271cc"}
Jul 30 18:23:59.438953 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:23:59.438624    2555 scope.go:117] "RemoveContainer" containerID="baf16852f97356370c0536681bc49083129a6ab8cbed3128915e956ace545c50"
Jul 30 18:23:59.449609 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:23:59.449581    2555 scope.go:117] "RemoveContainer" containerID="9e89f4e4612306f775edf27cdf9032854ceb6b1163894c85a826eba7b778dca3"
Jul 30 18:23:59.457546 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:23:59.457529    2555 scope.go:117] "RemoveContainer" containerID="0cd766236d6ca178706afacda90a2553519ebec627289b515772aea518c9e30a"
Jul 30 18:23:59.462572 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:23:59.462549    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ghgwx"]
Jul 30 18:23:59.466780 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:23:59.466758    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ghgwx"]
Jul 30 18:24:00.060050 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:24:00.060003    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e" path="/var/lib/kubelet/pods/8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e/volumes"
Jul 30 18:25:17.210126 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:25:17.210095    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:25:21.428841 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:25:21.428812    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:25:21.769451 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:25:21.769368    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:25:45.331939 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:25:45.331901    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:26:12.160590 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:26:12.160554    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:26:41.567328 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:26:41.567295    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:26:41.665936 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:26:41.665899    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:26:41.861549 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:26:41.861470    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:28:01.747920 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:28:01.747888    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:28:02.832999 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:28:02.832969    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:28:02.934261 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:28:02.934216    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:28:08.067854 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:28:08.067817    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:28:12.808237 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:28:12.808203    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:28:13.899049 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:28:13.898985    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:28:14.370399 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:28:14.370352    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:29:33.949002 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:29:33.948968    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:29:34.970097 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:29:34.970062    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:29:35.239879 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:29:35.239800    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:29:45.019857 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:29:45.019823    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:29:46.077132 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:29:46.077098    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:29:46.362123 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:29:46.362043    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:31:16.168798 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:31:16.168769    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:31:17.205889 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:31:17.205857    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:31:17.658444 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:31:17.658414    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:31:27.270233 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:31:27.270198    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:31:29.078941 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:31:29.078906    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:31:29.140750 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:31:29.140695    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:31:58.126083 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:31:58.125998    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:32:03.696351 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:03.696320    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:32:03.832785 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:03.832750    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mgfj9/must-gather-n98xm"]
Jul 30 18:32:03.833138 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:03.833126    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e" containerName="storage-initializer"
Jul 30 18:32:03.833190 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:03.833140    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e" containerName="storage-initializer"
Jul 30 18:32:03.833190 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:03.833149    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e" containerName="kserve-container"
Jul 30 18:32:03.833190 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:03.833154    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e" containerName="kserve-container"
Jul 30 18:32:03.833190 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:03.833164    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e" containerName="kube-rbac-proxy"
Jul 30 18:32:03.833190 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:03.833170    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e" containerName="kube-rbac-proxy"
Jul 30 18:32:03.833353 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:03.833243    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e" containerName="kserve-container"
Jul 30 18:32:03.833353 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:03.833252    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b1c74a5-6fb0-40a1-bf0d-e41e96c7746e" containerName="kube-rbac-proxy"
Jul 30 18:32:03.836290 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:03.836274    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mgfj9/must-gather-n98xm"
Jul 30 18:32:03.838786 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:03.838763    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mgfj9\"/\"openshift-service-ca.crt\""
Jul 30 18:32:03.838916 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:03.838764    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-mgfj9\"/\"default-dockercfg-rxz6h\""
Jul 30 18:32:03.838916 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:03.838778    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mgfj9\"/\"kube-root-ca.crt\""
Jul 30 18:32:03.845440 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:03.845401    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mgfj9/must-gather-n98xm"]
Jul 30 18:32:03.860920 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:03.860896    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:32:03.961781 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:03.961694    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:32:03.996836 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:03.996803    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/aab04bf5-8a60-4870-b01c-663c983fb844-must-gather-output\") pod \"must-gather-n98xm\" (UID: \"aab04bf5-8a60-4870-b01c-663c983fb844\") " pod="openshift-must-gather-mgfj9/must-gather-n98xm"
Jul 30 18:32:03.996836 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:03.996837    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6m4n\" (UniqueName: \"kubernetes.io/projected/aab04bf5-8a60-4870-b01c-663c983fb844-kube-api-access-t6m4n\") pod \"must-gather-n98xm\" (UID: \"aab04bf5-8a60-4870-b01c-663c983fb844\") " pod="openshift-must-gather-mgfj9/must-gather-n98xm"
Jul 30 18:32:04.098262 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:04.098219    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/aab04bf5-8a60-4870-b01c-663c983fb844-must-gather-output\") pod \"must-gather-n98xm\" (UID: \"aab04bf5-8a60-4870-b01c-663c983fb844\") " pod="openshift-must-gather-mgfj9/must-gather-n98xm"
Jul 30 18:32:04.098262 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:04.098260    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6m4n\" (UniqueName: \"kubernetes.io/projected/aab04bf5-8a60-4870-b01c-663c983fb844-kube-api-access-t6m4n\") pod \"must-gather-n98xm\" (UID: \"aab04bf5-8a60-4870-b01c-663c983fb844\") " pod="openshift-must-gather-mgfj9/must-gather-n98xm"
Jul 30 18:32:04.098588 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:04.098570    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/aab04bf5-8a60-4870-b01c-663c983fb844-must-gather-output\") pod \"must-gather-n98xm\" (UID: \"aab04bf5-8a60-4870-b01c-663c983fb844\") " pod="openshift-must-gather-mgfj9/must-gather-n98xm"
Jul 30 18:32:04.107819 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:04.107797    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6m4n\" (UniqueName: \"kubernetes.io/projected/aab04bf5-8a60-4870-b01c-663c983fb844-kube-api-access-t6m4n\") pod \"must-gather-n98xm\" (UID: \"aab04bf5-8a60-4870-b01c-663c983fb844\") " pod="openshift-must-gather-mgfj9/must-gather-n98xm"
Jul 30 18:32:04.157586 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:04.157543    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mgfj9/must-gather-n98xm"
Jul 30 18:32:04.292479 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:04.292449    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mgfj9/must-gather-n98xm"]
Jul 30 18:32:04.296248 ip-10-0-143-190 kubenswrapper[2555]: W0730 18:32:04.296220    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaab04bf5_8a60_4870_b01c_663c983fb844.slice/crio-31aa4b62979f780c878eecb2bc985ba13e70587f117d53f885a6a25a5049f9be WatchSource:0}: Error finding container 31aa4b62979f780c878eecb2bc985ba13e70587f117d53f885a6a25a5049f9be: Status 404 returned error can't find the container with id 31aa4b62979f780c878eecb2bc985ba13e70587f117d53f885a6a25a5049f9be
Jul 30 18:32:04.297979 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:04.297962    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 30 18:32:04.654479 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:04.654446    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mgfj9/must-gather-n98xm" event={"ID":"aab04bf5-8a60-4870-b01c-663c983fb844","Type":"ContainerStarted","Data":"31aa4b62979f780c878eecb2bc985ba13e70587f117d53f885a6a25a5049f9be"}
Jul 30 18:32:05.661689 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:05.661649    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mgfj9/must-gather-n98xm" event={"ID":"aab04bf5-8a60-4870-b01c-663c983fb844","Type":"ContainerStarted","Data":"4278ae1cc87ec3dee89d2deb284153c117c7a1c89965403b847cabad615f4c93"}
Jul 30 18:32:05.662101 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:05.661699    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mgfj9/must-gather-n98xm" event={"ID":"aab04bf5-8a60-4870-b01c-663c983fb844","Type":"ContainerStarted","Data":"326e66032faf65274acc0fa9a6116b6534298591e3e0bf259767ac417f55995c"}
Jul 30 18:32:05.678671 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:05.678625    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-mgfj9/must-gather-n98xm" podStartSLOduration=1.778059971 podStartE2EDuration="2.678610199s" totalImagesPullingTime="900.550228ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 18:32:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 18:32:05.677125288 +0000 UTC m=+5500.164492689" watchObservedRunningTime="2026-07-30 18:32:05.678610199 +0000 UTC m=+5500.165977599"
Jul 30 18:32:07.882819 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:07.882789    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-9hlj5_51afa6bb-724f-47ab-b47b-36dc8a61ca25/global-pull-secret-syncer/0.log"
Jul 30 18:32:07.994666 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:07.994628    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log"
Jul 30 18:32:08.184142 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:08.184071    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-190.ec2.internal_041d73f2cda3ec42dcee6249e2df6511/haproxy/0.log"
Jul 30 18:32:09.529607 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:09.529561    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:32:09.586894 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:09.586850    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8lckj_9485af8e-c7d9-4b8a-ab94-aa3b30415381/konnectivity-agent/0.log" limit=1024
Jul 30 18:32:11.978931 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:11.978902    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c66554ef-e555-43c5-b2eb-cdaa50f6e3d4/alertmanager/0.log"
Jul 30 18:32:12.011634 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:12.011610    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c66554ef-e555-43c5-b2eb-cdaa50f6e3d4/config-reloader/0.log"
Jul 30 18:32:12.078257 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:12.078226    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c66554ef-e555-43c5-b2eb-cdaa50f6e3d4/kube-rbac-proxy-web/0.log"
Jul 30 18:32:12.101689 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:12.101658    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c66554ef-e555-43c5-b2eb-cdaa50f6e3d4/kube-rbac-proxy/0.log"
Jul 30 18:32:12.123884 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:12.123859    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c66554ef-e555-43c5-b2eb-cdaa50f6e3d4/kube-rbac-proxy-metric/0.log"
Jul 30 18:32:12.148026 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:12.147990    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c66554ef-e555-43c5-b2eb-cdaa50f6e3d4/prom-label-proxy/0.log"
Jul 30 18:32:12.206773 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:12.206746    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c66554ef-e555-43c5-b2eb-cdaa50f6e3d4/init-config-reloader/0.log"
Jul 30 18:32:12.382512 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:12.382473    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7545856648-6x6pd_42318d15-ab44-4938-930f-ee2961a6d00b/metrics-server/0.log"
Jul 30 18:32:12.422607 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:12.422580    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-dlr24_411dc2d0-e97f-4574-abe7-1e06f22653f9/monitoring-plugin/0.log"
Jul 30 18:32:12.533081 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:12.533048    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9q5tr_ed87dbe4-de48-4ff6-bb84-cac7b1719661/node-exporter/0.log"
Jul 30 18:32:12.555596 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:12.555562    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9q5tr_ed87dbe4-de48-4ff6-bb84-cac7b1719661/kube-rbac-proxy/0.log"
Jul 30 18:32:12.578785 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:12.578755    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9q5tr_ed87dbe4-de48-4ff6-bb84-cac7b1719661/init-textfile/0.log"
Jul 30 18:32:12.688969 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:12.688876    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-qvrdk_45cb4db9-0f93-4fcb-80c5-20ff7c527aab/kube-rbac-proxy-main/0.log"
Jul 30 18:32:12.709659 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:12.709632    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-qvrdk_45cb4db9-0f93-4fcb-80c5-20ff7c527aab/kube-rbac-proxy-self/0.log"
Jul 30 18:32:12.733463 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:12.733437    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-qvrdk_45cb4db9-0f93-4fcb-80c5-20ff7c527aab/openshift-state-metrics/0.log"
Jul 30 18:32:12.956822 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:12.956736    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-7s9bw_14057a67-fd8e-40e9-91c9-3ff562752bbf/prometheus-operator/0.log"
Jul 30 18:32:12.976456 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:12.976430    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-7s9bw_14057a67-fd8e-40e9-91c9-3ff562752bbf/kube-rbac-proxy/0.log"
Jul 30 18:32:13.031685 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:13.031657    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7679956457-564g9_4e9daaaf-67dc-4b6c-bae1-2e2d092d4e5d/telemeter-client/0.log"
Jul 30 18:32:13.062918 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:13.062850    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7679956457-564g9_4e9daaaf-67dc-4b6c-bae1-2e2d092d4e5d/reload/0.log"
Jul 30 18:32:13.083740 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:13.083711    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7679956457-564g9_4e9daaaf-67dc-4b6c-bae1-2e2d092d4e5d/kube-rbac-proxy/0.log"
Jul 30 18:32:13.117814 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:13.117786    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6448464666-hls4h_266b5284-fb72-46c4-a694-37535e997cd3/thanos-query/0.log"
Jul 30 18:32:13.140363 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:13.140319    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6448464666-hls4h_266b5284-fb72-46c4-a694-37535e997cd3/kube-rbac-proxy-web/0.log"
Jul 30 18:32:13.160461 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:13.160401    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6448464666-hls4h_266b5284-fb72-46c4-a694-37535e997cd3/kube-rbac-proxy/0.log"
Jul 30 18:32:13.186979 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:13.186949    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6448464666-hls4h_266b5284-fb72-46c4-a694-37535e997cd3/prom-label-proxy/0.log"
Jul 30 18:32:13.207483 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:13.207406    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6448464666-hls4h_266b5284-fb72-46c4-a694-37535e997cd3/kube-rbac-proxy-rules/0.log"
Jul 30 18:32:13.228590 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:13.228495    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6448464666-hls4h_266b5284-fb72-46c4-a694-37535e997cd3/kube-rbac-proxy-metrics/0.log"
Jul 30 18:32:15.689356 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:15.689330    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-857dfd46db-84ghf_18d3d3a8-db9c-4a25-bf94-535f65ad2ea6/console/0.log"
Jul 30 18:32:15.732057 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:15.732027    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-lgsc2_7bb3cf69-d970-4823-b2f4-f9f726efaf9d/download-server/0.log"
Jul 30 18:32:16.953300 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:16.953264    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mgfj9/perf-node-gather-daemonset-nswfz"]
Jul 30 18:32:16.960073 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:16.960045    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mgfj9/perf-node-gather-daemonset-nswfz"
Jul 30 18:32:16.966417 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:16.966385    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mgfj9/perf-node-gather-daemonset-nswfz"]
Jul 30 18:32:17.039788 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:17.039741    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d57c7846-840f-4c02-a74b-5046149ba901-lib-modules\") pod \"perf-node-gather-daemonset-nswfz\" (UID: \"d57c7846-840f-4c02-a74b-5046149ba901\") " pod="openshift-must-gather-mgfj9/perf-node-gather-daemonset-nswfz"
Jul 30 18:32:17.039983 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:17.039808    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d57c7846-840f-4c02-a74b-5046149ba901-podres\") pod \"perf-node-gather-daemonset-nswfz\" (UID: \"d57c7846-840f-4c02-a74b-5046149ba901\") " pod="openshift-must-gather-mgfj9/perf-node-gather-daemonset-nswfz"
Jul 30 18:32:17.039983 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:17.039835    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4blv\" (UniqueName: \"kubernetes.io/projected/d57c7846-840f-4c02-a74b-5046149ba901-kube-api-access-f4blv\") pod \"perf-node-gather-daemonset-nswfz\" (UID: \"d57c7846-840f-4c02-a74b-5046149ba901\") " pod="openshift-must-gather-mgfj9/perf-node-gather-daemonset-nswfz"
Jul 30 18:32:17.039983 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:17.039921    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d57c7846-840f-4c02-a74b-5046149ba901-proc\") pod \"perf-node-gather-daemonset-nswfz\" (UID: \"d57c7846-840f-4c02-a74b-5046149ba901\") " pod="openshift-must-gather-mgfj9/perf-node-gather-daemonset-nswfz"
Jul 30 18:32:17.040183 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:17.039987    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d57c7846-840f-4c02-a74b-5046149ba901-sys\") pod \"perf-node-gather-daemonset-nswfz\" (UID: \"d57c7846-840f-4c02-a74b-5046149ba901\") " pod="openshift-must-gather-mgfj9/perf-node-gather-daemonset-nswfz"
Jul 30 18:32:17.140578 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:17.140538    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d57c7846-840f-4c02-a74b-5046149ba901-podres\") pod \"perf-node-gather-daemonset-nswfz\" (UID: \"d57c7846-840f-4c02-a74b-5046149ba901\") " pod="openshift-must-gather-mgfj9/perf-node-gather-daemonset-nswfz"
Jul 30 18:32:17.140771 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:17.140588    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4blv\" (UniqueName: \"kubernetes.io/projected/d57c7846-840f-4c02-a74b-5046149ba901-kube-api-access-f4blv\") pod \"perf-node-gather-daemonset-nswfz\" (UID: \"d57c7846-840f-4c02-a74b-5046149ba901\") " pod="openshift-must-gather-mgfj9/perf-node-gather-daemonset-nswfz"
Jul 30 18:32:17.140771 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:17.140699    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d57c7846-840f-4c02-a74b-5046149ba901-proc\") pod \"perf-node-gather-daemonset-nswfz\" (UID: \"d57c7846-840f-4c02-a74b-5046149ba901\") " pod="openshift-must-gather-mgfj9/perf-node-gather-daemonset-nswfz"
Jul 30 18:32:17.140771 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:17.140708    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d57c7846-840f-4c02-a74b-5046149ba901-podres\") pod \"perf-node-gather-daemonset-nswfz\" (UID: \"d57c7846-840f-4c02-a74b-5046149ba901\") " pod="openshift-must-gather-mgfj9/perf-node-gather-daemonset-nswfz"
Jul 30 18:32:17.140958 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:17.140771    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d57c7846-840f-4c02-a74b-5046149ba901-proc\") pod \"perf-node-gather-daemonset-nswfz\" (UID: \"d57c7846-840f-4c02-a74b-5046149ba901\") " pod="openshift-must-gather-mgfj9/perf-node-gather-daemonset-nswfz"
Jul 30 18:32:17.140958 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:17.140894    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d57c7846-840f-4c02-a74b-5046149ba901-sys\") pod \"perf-node-gather-daemonset-nswfz\" (UID: \"d57c7846-840f-4c02-a74b-5046149ba901\") " pod="openshift-must-gather-mgfj9/perf-node-gather-daemonset-nswfz"
Jul 30 18:32:17.141107 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:17.140972    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d57c7846-840f-4c02-a74b-5046149ba901-lib-modules\") pod \"perf-node-gather-daemonset-nswfz\" (UID: \"d57c7846-840f-4c02-a74b-5046149ba901\") " pod="openshift-must-gather-mgfj9/perf-node-gather-daemonset-nswfz"
Jul 30 18:32:17.141107 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:17.140983    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d57c7846-840f-4c02-a74b-5046149ba901-sys\") pod \"perf-node-gather-daemonset-nswfz\" (UID: \"d57c7846-840f-4c02-a74b-5046149ba901\") " pod="openshift-must-gather-mgfj9/perf-node-gather-daemonset-nswfz"
Jul 30 18:32:17.141107 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:17.141094    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d57c7846-840f-4c02-a74b-5046149ba901-lib-modules\") pod \"perf-node-gather-daemonset-nswfz\" (UID: \"d57c7846-840f-4c02-a74b-5046149ba901\") " pod="openshift-must-gather-mgfj9/perf-node-gather-daemonset-nswfz"
Jul 30 18:32:17.149764 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:17.149736    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4blv\" (UniqueName: \"kubernetes.io/projected/d57c7846-840f-4c02-a74b-5046149ba901-kube-api-access-f4blv\") pod \"perf-node-gather-daemonset-nswfz\" (UID: \"d57c7846-840f-4c02-a74b-5046149ba901\") " pod="openshift-must-gather-mgfj9/perf-node-gather-daemonset-nswfz"
Jul 30 18:32:17.223146 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:17.223074    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-l9zx6_13abbab1-dfcf-4102-9a67-54f9b636e1f3/dns/0.log"
Jul 30 18:32:17.242512 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:17.242433    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-l9zx6_13abbab1-dfcf-4102-9a67-54f9b636e1f3/kube-rbac-proxy/0.log"
Jul 30 18:32:17.273192 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:17.273147    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mgfj9/perf-node-gather-daemonset-nswfz"
Jul 30 18:32:17.285475 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:17.285413    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-97k9j_3534e135-e217-49b8-8866-8cd33104b77e/dns-node-resolver/0.log"
Jul 30 18:32:17.447676 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:17.447645    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mgfj9/perf-node-gather-daemonset-nswfz"]
Jul 30 18:32:17.451583 ip-10-0-143-190 kubenswrapper[2555]: W0730 18:32:17.451546    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd57c7846_840f_4c02_a74b_5046149ba901.slice/crio-8216a525e7d064f8284ff392703a5739c5e4ff48446c02c457007791e9979835 WatchSource:0}: Error finding container 8216a525e7d064f8284ff392703a5739c5e4ff48446c02c457007791e9979835: Status 404 returned error can't find the container with id 8216a525e7d064f8284ff392703a5739c5e4ff48446c02c457007791e9979835
Jul 30 18:32:17.741681 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:17.741644    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mgfj9/perf-node-gather-daemonset-nswfz" event={"ID":"d57c7846-840f-4c02-a74b-5046149ba901","Type":"ContainerStarted","Data":"87fe71a4ba21aa53c4e7b631e877d805519fc593747d8a1272991d61925f576f"}
Jul 30 18:32:17.741862 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:17.741707    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mgfj9/perf-node-gather-daemonset-nswfz" event={"ID":"d57c7846-840f-4c02-a74b-5046149ba901","Type":"ContainerStarted","Data":"8216a525e7d064f8284ff392703a5739c5e4ff48446c02c457007791e9979835"}
Jul 30 18:32:17.741862 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:17.741786    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-mgfj9/perf-node-gather-daemonset-nswfz"
Jul 30 18:32:17.763848 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:17.763741    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-mgfj9/perf-node-gather-daemonset-nswfz" podStartSLOduration=1.763724329 podStartE2EDuration="1.763724329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 18:32:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 18:32:17.761347217 +0000 UTC m=+5512.248714619" watchObservedRunningTime="2026-07-30 18:32:17.763724329 +0000 UTC m=+5512.251091729"
Jul 30 18:32:17.899255 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:17.899226    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xglts_97f5f5b8-ce32-41cd-9b5c-c94f2ae6bbfe/node-ca/0.log"
Jul 30 18:32:18.984687 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:18.984664    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-g5pvs_ac14b3bf-789c-4419-a46b-55fab58dcfd8/serve-healthcheck-canary/0.log"
Jul 30 18:32:19.539784 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:19.539759    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tkbtl_a340f4e6-daaf-450e-b880-a3658b1296ef/kube-rbac-proxy/0.log"
Jul 30 18:32:19.560306 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:19.560282    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tkbtl_a340f4e6-daaf-450e-b880-a3658b1296ef/exporter/0.log"
Jul 30 18:32:19.579572 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:19.579531    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tkbtl_a340f4e6-daaf-450e-b880-a3658b1296ef/extractor/0.log"
Jul 30 18:32:21.893635 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:21.893602    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-8677446b84-rz8nz_6ed65973-197a-4225-b161-4f847067a7c1/manager/0.log"
Jul 30 18:32:21.946361 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:21.946337    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-kvr8r_0d30e976-fa6b-44af-9a2a-5d8433a3cd84/server/0.log"
Jul 30 18:32:22.291850 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:22.291818    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-8588677cfb-zbjs8_79f9bec6-db1a-4830-b663-2f3435132f67/manager/0.log"
Jul 30 18:32:22.481650 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:22.481625    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-jv5m4_d4e163c0-ecd2-467e-b228-5a97deba9f0e/seaweedfs/0.log"
Jul 30 18:32:22.505849 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:22.505823    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-custom-8b44d767d-rwg8w_1a9ee89d-0575-405a-8d22-b0e4070283f6/seaweedfs-tls-custom/0.log"
Jul 30 18:32:22.529343 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:22.529288    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-serving-67b7477c6d-n52p8_af454a5f-6e99-4da4-98a8-ea7cde369acc/seaweedfs-tls-serving/0.log"
Jul 30 18:32:23.756607 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:23.756579    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-mgfj9/perf-node-gather-daemonset-nswfz"
Jul 30 18:32:27.944141 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:27.944112    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qbp2g_9fcdb3a5-74bc-44be-993f-fbc90e5db202/kube-multus-additional-cni-plugins/0.log"
Jul 30 18:32:27.965166 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:27.965143    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qbp2g_9fcdb3a5-74bc-44be-993f-fbc90e5db202/egress-router-binary-copy/0.log"
Jul 30 18:32:27.989882 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:27.989852    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qbp2g_9fcdb3a5-74bc-44be-993f-fbc90e5db202/cni-plugins/0.log"
Jul 30 18:32:28.009890 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:28.009863    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qbp2g_9fcdb3a5-74bc-44be-993f-fbc90e5db202/bond-cni-plugin/0.log"
Jul 30 18:32:28.030160 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:28.030128    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qbp2g_9fcdb3a5-74bc-44be-993f-fbc90e5db202/routeoverride-cni/0.log"
Jul 30 18:32:28.050874 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:28.050818    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qbp2g_9fcdb3a5-74bc-44be-993f-fbc90e5db202/whereabouts-cni-bincopy/0.log"
Jul 30 18:32:28.071419 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:28.071392    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qbp2g_9fcdb3a5-74bc-44be-993f-fbc90e5db202/whereabouts-cni/0.log"
Jul 30 18:32:28.317066 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:28.317041    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-nfpq4_a2093549-0eca-49bd-96d3-2e0c01a4c19c/kube-multus/0.log"
Jul 30 18:32:28.336777 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:28.336753    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-55hfh_e85f9d71-fe3d-49d8-93e2-e4ac101603c7/network-metrics-daemon/0.log"
Jul 30 18:32:28.355204 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:28.355178    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-55hfh_e85f9d71-fe3d-49d8-93e2-e4ac101603c7/kube-rbac-proxy/0.log"
Jul 30 18:32:29.614887 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:29.614856    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t87pw_e1dcce1f-b59e-47cc-9015-429f8a7858cd/ovn-controller/0.log"
Jul 30 18:32:29.630981 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:29.630955    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t87pw_e1dcce1f-b59e-47cc-9015-429f8a7858cd/ovn-acl-logging/0.log"
Jul 30 18:32:29.652870 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:29.652844    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t87pw_e1dcce1f-b59e-47cc-9015-429f8a7858cd/kube-rbac-proxy-node/0.log"
Jul 30 18:32:29.671803 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:29.671777    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t87pw_e1dcce1f-b59e-47cc-9015-429f8a7858cd/kube-rbac-proxy-ovn-metrics/0.log"
Jul 30 18:32:29.690064 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:29.690032    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t87pw_e1dcce1f-b59e-47cc-9015-429f8a7858cd/northd/0.log"
Jul 30 18:32:29.709533 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:29.709498    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t87pw_e1dcce1f-b59e-47cc-9015-429f8a7858cd/nbdb/0.log"
Jul 30 18:32:29.732482 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:29.732459    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t87pw_e1dcce1f-b59e-47cc-9015-429f8a7858cd/sbdb/0.log"
Jul 30 18:32:29.783726 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:29.783697    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t87pw_e1dcce1f-b59e-47cc-9015-429f8a7858cd/ovnkube-controller/0.log"
Jul 30 18:32:30.827394 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:30.827367    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-qvgrv_fc0a3112-260e-47c8-87fd-bf3666ea17d0/network-check-target-container/0.log"
Jul 30 18:32:31.700387 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:31.700361    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-2tdqf_ed0f2ced-c500-4288-ac35-3e5599c9d25f/iptables-alerter/0.log"
Jul 30 18:32:32.390248 ip-10-0-143-190 kubenswrapper[2555]: I0730 18:32:32.390221    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-bncwx_80fefbe4-044f-4c09-b4ae-4bcc3d5684d6/tuned/0.log"