Jul 28 10:07:39.296451 ip-10-0-132-110 systemd[1]: Starting Kubernetes Kubelet...
Jul 28 10:07:39.671769 ip-10-0-132-110 kubenswrapper[2583]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 10:07:39.671769 ip-10-0-132-110 kubenswrapper[2583]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 28 10:07:39.671769 ip-10-0-132-110 kubenswrapper[2583]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 10:07:39.671769 ip-10-0-132-110 kubenswrapper[2583]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 28 10:07:39.671769 ip-10-0-132-110 kubenswrapper[2583]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 10:07:39.672452 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.672354    2583 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 28 10:07:39.677699 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677684    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 10:07:39.677699 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677700    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 10:07:39.677766 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677704    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 10:07:39.677766 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677707    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 10:07:39.677766 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677710    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 10:07:39.677766 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677713    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 10:07:39.677766 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677716    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 10:07:39.677766 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677718    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 10:07:39.677766 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677721    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 10:07:39.677766 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677723    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 10:07:39.677766 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677726    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 10:07:39.677766 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677729    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 10:07:39.677766 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677732    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 10:07:39.677766 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677734    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 10:07:39.677766 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677737    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 10:07:39.677766 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677739    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 10:07:39.677766 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677742    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 10:07:39.677766 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677744    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 10:07:39.677766 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677747    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 10:07:39.677766 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677750    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 10:07:39.677766 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677753    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 10:07:39.677766 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677755    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 10:07:39.678252 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677758    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 10:07:39.678252 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677760    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 10:07:39.678252 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677763    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 10:07:39.678252 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677765    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 10:07:39.678252 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677768    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 10:07:39.678252 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677771    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 10:07:39.678252 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677774    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 10:07:39.678252 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677777    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 10:07:39.678252 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677780    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 10:07:39.678252 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677784    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 10:07:39.678252 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677788    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 10:07:39.678252 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677791    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 10:07:39.678252 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677794    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 10:07:39.678252 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677797    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 10:07:39.678252 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677808    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 10:07:39.678252 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677812    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 10:07:39.678252 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677816    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 10:07:39.678252 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677818    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 10:07:39.678252 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677821    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 10:07:39.678710 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677824    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 10:07:39.678710 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677826    2583 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 10:07:39.678710 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677829    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 10:07:39.678710 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677831    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 10:07:39.678710 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677834    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 10:07:39.678710 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677837    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 10:07:39.678710 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677839    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 10:07:39.678710 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677842    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 10:07:39.678710 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677844    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 10:07:39.678710 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677847    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 10:07:39.678710 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677849    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 10:07:39.678710 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677852    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 10:07:39.678710 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677855    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 10:07:39.678710 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677857    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 10:07:39.678710 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677862    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 10:07:39.678710 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677865    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 10:07:39.678710 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677867    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 10:07:39.678710 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677870    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 10:07:39.678710 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677872    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 10:07:39.678710 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677875    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 10:07:39.679256 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677878    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 10:07:39.679256 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677881    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 10:07:39.679256 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677883    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 10:07:39.679256 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677886    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 10:07:39.679256 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677888    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 10:07:39.679256 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677891    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 10:07:39.679256 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677894    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 10:07:39.679256 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677897    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 10:07:39.679256 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677899    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 10:07:39.679256 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677902    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 10:07:39.679256 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677905    2583 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 10:07:39.679256 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677907    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 10:07:39.679256 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677910    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 10:07:39.679256 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677912    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 10:07:39.679256 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677915    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 10:07:39.679256 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677917    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 10:07:39.679256 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677920    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 10:07:39.679256 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677923    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 10:07:39.679256 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677926    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 10:07:39.679256 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677929    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 10:07:39.679762 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677931    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 10:07:39.679762 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677933    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 10:07:39.679762 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677936    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 10:07:39.679762 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677938    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 10:07:39.679762 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677941    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 10:07:39.679762 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677945    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 10:07:39.679762 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677948    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 10:07:39.679762 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677952    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 10:07:39.679762 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677955    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 10:07:39.679762 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677957    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 10:07:39.679762 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677960    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 10:07:39.679762 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677963    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 10:07:39.679762 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677966    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 10:07:39.679762 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677969    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 10:07:39.679762 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677972    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 10:07:39.679762 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677975    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 10:07:39.679762 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677978    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 10:07:39.679762 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.677980    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 10:07:39.679762 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678668    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 10:07:39.680300 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678675    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 10:07:39.680300 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678678    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 10:07:39.680300 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678681    2583 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 10:07:39.680300 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678684    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 10:07:39.680300 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678686    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 10:07:39.680300 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678689    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 10:07:39.680300 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678692    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 10:07:39.680300 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678696    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 10:07:39.680300 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678700    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 10:07:39.680300 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678703    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 10:07:39.680300 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678706    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 10:07:39.680300 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678709    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 10:07:39.680300 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678712    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 10:07:39.680300 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678715    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 10:07:39.680300 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678718    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 10:07:39.680300 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678721    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 10:07:39.680300 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678723    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 10:07:39.680300 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678726    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 10:07:39.680300 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678729    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 10:07:39.680300 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678733    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 10:07:39.680796 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678735    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 10:07:39.680796 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678738    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 10:07:39.680796 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678741    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 10:07:39.680796 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678743    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 10:07:39.680796 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678746    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 10:07:39.680796 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678748    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 10:07:39.680796 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678751    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 10:07:39.680796 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678754    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 10:07:39.680796 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678756    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 10:07:39.680796 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678759    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 10:07:39.680796 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678761    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 10:07:39.680796 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678764    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 10:07:39.680796 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678767    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 10:07:39.680796 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678769    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 10:07:39.680796 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678772    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 10:07:39.680796 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678775    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 10:07:39.680796 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678779    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 10:07:39.680796 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678782    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 10:07:39.680796 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678784    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 10:07:39.680796 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678787    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 10:07:39.681308 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678789    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 10:07:39.681308 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678791    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 10:07:39.681308 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678795    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 10:07:39.681308 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678797    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 10:07:39.681308 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678799    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 10:07:39.681308 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678802    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 10:07:39.681308 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678804    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 10:07:39.681308 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678807    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 10:07:39.681308 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678810    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 10:07:39.681308 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678812    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 10:07:39.681308 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678817    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 10:07:39.681308 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678822    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 10:07:39.681308 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678826    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 10:07:39.681308 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678829    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 10:07:39.681308 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678832    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 10:07:39.681308 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678835    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 10:07:39.681308 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678837    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 10:07:39.681308 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678840    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 10:07:39.681308 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678843    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 10:07:39.681896 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678846    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 10:07:39.681896 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678849    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 10:07:39.681896 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678852    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 10:07:39.681896 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678854    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 10:07:39.681896 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678857    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 10:07:39.681896 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678860    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 10:07:39.681896 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678862    2583 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 10:07:39.681896 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678865    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 10:07:39.681896 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678867    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 10:07:39.681896 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678870    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 10:07:39.681896 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678872    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 10:07:39.681896 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678875    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 10:07:39.681896 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678877    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 10:07:39.681896 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678880    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 10:07:39.681896 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678882    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 10:07:39.681896 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678885    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 10:07:39.681896 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678887    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 10:07:39.681896 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678889    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 10:07:39.681896 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678892    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 10:07:39.681896 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678894    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 10:07:39.682674 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678897    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 10:07:39.682674 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678899    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 10:07:39.682674 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678902    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 10:07:39.682674 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678906    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 10:07:39.682674 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678909    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 10:07:39.682674 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678913    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 10:07:39.682674 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678916    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 10:07:39.682674 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678919    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 10:07:39.682674 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678921    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 10:07:39.682674 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678923    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 10:07:39.682674 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678926    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 10:07:39.682674 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678929    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 10:07:39.682674 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678932    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 10:07:39.682674 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678934    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 10:07:39.682674 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678937    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 10:07:39.682674 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678939    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 10:07:39.682674 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678941    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 10:07:39.682674 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678944    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 10:07:39.682674 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.678946    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 10:07:39.682674 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679378    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 10:07:39.683412 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679390    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 10:07:39.683412 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679393    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 10:07:39.683412 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679396    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 10:07:39.683412 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679399    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 10:07:39.683412 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679402    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 10:07:39.683412 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679405    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 10:07:39.683412 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679407    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 10:07:39.683412 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679410    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 10:07:39.683412 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679413    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 10:07:39.683412 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679415    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 10:07:39.683412 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679418    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 10:07:39.683412 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679420    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 10:07:39.683412 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679425    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 10:07:39.683412 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679429    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 10:07:39.683412 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679432    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 10:07:39.683412 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679435    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 10:07:39.683412 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679438    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 10:07:39.683412 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679440    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 10:07:39.683412 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679443    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 10:07:39.683888 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679446    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 10:07:39.683888 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679448    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 10:07:39.683888 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679451    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 10:07:39.683888 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679471    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 10:07:39.683888 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679475    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 10:07:39.683888 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679479    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 10:07:39.683888 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679483    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 10:07:39.683888 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679486    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 10:07:39.683888 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679488    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 10:07:39.683888 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679492    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 10:07:39.683888 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679495    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 10:07:39.683888 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679497    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 10:07:39.683888 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679500    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 10:07:39.683888 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679503    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 10:07:39.683888 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679506    2583 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 10:07:39.683888 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679508    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 10:07:39.683888 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679511    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 10:07:39.683888 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679514    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 10:07:39.683888 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679517    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 10:07:39.684456 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679519    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 10:07:39.684456 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679522    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 10:07:39.684456 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679524    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 10:07:39.684456 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679527    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 10:07:39.684456 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679529    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 10:07:39.684456 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679532    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 10:07:39.684456 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679534    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 10:07:39.684456 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679537    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 10:07:39.684456 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679539    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 10:07:39.684456 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679542    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 10:07:39.684456 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679544    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 10:07:39.684456 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679548    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 10:07:39.684456 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679552    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 10:07:39.684456 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679555    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 10:07:39.684456 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679557    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 10:07:39.684456 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679560    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 10:07:39.684456 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679564    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 10:07:39.684456 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679567    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 10:07:39.684456 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679571    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 10:07:39.684956 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679574    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 10:07:39.684956 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679576    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 10:07:39.684956 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679579    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 10:07:39.684956 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679582    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 10:07:39.684956 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679585    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 10:07:39.684956 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679588    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 10:07:39.684956 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679590    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 10:07:39.684956 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679596    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 10:07:39.684956 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679598    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 10:07:39.684956 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679601    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 10:07:39.684956 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679604    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 10:07:39.684956 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679607    2583 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 10:07:39.684956 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679609    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 10:07:39.684956 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679612    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 10:07:39.684956 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679615    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 10:07:39.684956 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679617    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 10:07:39.684956 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679620    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 10:07:39.684956 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679622    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 10:07:39.684956 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679625    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 10:07:39.684956 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679628    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 10:07:39.685476 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679630    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 10:07:39.685476 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679633    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 10:07:39.685476 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679637    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 10:07:39.685476 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679639    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 10:07:39.685476 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679642    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 10:07:39.685476 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679644    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 10:07:39.685476 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679647    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 10:07:39.685476 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679649    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 10:07:39.685476 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679652    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 10:07:39.685476 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679654    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 10:07:39.685476 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679657    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 10:07:39.685476 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679660    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 10:07:39.685476 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679663    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 10:07:39.685476 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679665    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 10:07:39.685476 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679668    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 10:07:39.685476 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679671    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 10:07:39.685476 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679673    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 10:07:39.685476 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679677    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 10:07:39.685476 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679679    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 10:07:39.685476 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679684    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 10:07:39.685958 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.679687    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 10:07:39.685958 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.679748    2583 flags.go:64] FLAG: --address="0.0.0.0"
Jul 28 10:07:39.685958 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.679755    2583 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 28 10:07:39.685958 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.679760    2583 flags.go:64] FLAG: --anonymous-auth="true"
Jul 28 10:07:39.685958 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.679764    2583 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 28 10:07:39.685958 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.679769    2583 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 28 10:07:39.685958 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.679772    2583 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 28 10:07:39.685958 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.679776    2583 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 28 10:07:39.685958 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.679780    2583 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 28 10:07:39.685958 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.679783    2583 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 28 10:07:39.685958 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.679786    2583 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 28 10:07:39.685958 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.679790    2583 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 28 10:07:39.685958 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.679793    2583 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 28 10:07:39.685958 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.679796    2583 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 28 10:07:39.685958 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.679799    2583 flags.go:64] FLAG: --cgroup-root=""
Jul 28 10:07:39.685958 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.679802    2583 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 28 10:07:39.685958 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.679805    2583 flags.go:64] FLAG: --client-ca-file=""
Jul 28 10:07:39.685958 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.679807    2583 flags.go:64] FLAG: --cloud-provider="external"
Jul 28 10:07:39.685958 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.679810    2583 flags.go:64] FLAG: --cluster-dns="[]"
Jul 28 10:07:39.685958 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.679814    2583 flags.go:64] FLAG: --cluster-domain=""
Jul 28 10:07:39.685958 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.679817    2583 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 28 10:07:39.685958 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.679820    2583 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 28 10:07:39.685958 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.679823    2583 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 28 10:07:39.685958 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.679828    2583 flags.go:64] FLAG: --container-log-max-files="5"
Jul 28 10:07:39.686558 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.679832    2583 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 28 10:07:39.686558 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.679834    2583 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 28 10:07:39.686558 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.679837    2583 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 28 10:07:39.686558 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.679841    2583 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 28 10:07:39.686558 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.679844    2583 flags.go:64] FLAG: --contention-profiling="false"
Jul 28 10:07:39.686558 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.679847    2583 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 28 10:07:39.686558 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.679850    2583 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 28 10:07:39.686558 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.679854    2583 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 28 10:07:39.686558 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.679857    2583 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 28 10:07:39.686558 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.679861    2583 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 28 10:07:39.686558 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.679864    2583 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 28 10:07:39.686558 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.679867    2583 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 28 10:07:39.686558 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.679870    2583 flags.go:64] FLAG: --enable-load-reader="false"
Jul 28 10:07:39.686558 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.679873    2583 flags.go:64] FLAG: --enable-server="true"
Jul 28 10:07:39.686558 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.679876    2583 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 28 10:07:39.686558 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.679879    2583 flags.go:64] FLAG: --event-burst="100"
Jul 28 10:07:39.686558 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.679883    2583 flags.go:64] FLAG: --event-qps="50"
Jul 28 10:07:39.686558 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.679886    2583 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 28 10:07:39.686558 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.679889    2583 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 28 10:07:39.686558 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.679892    2583 flags.go:64] FLAG: --eviction-hard=""
Jul 28 10:07:39.686558 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.679896    2583 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 28 10:07:39.686558 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.679898    2583 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 28 10:07:39.686558 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.679901    2583 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 28 10:07:39.686558 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.679904    2583 flags.go:64] FLAG: --eviction-soft=""
Jul 28 10:07:39.686558 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.679907    2583 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 28 10:07:39.687210 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.679910    2583 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 28 10:07:39.687210 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.679913    2583 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 28 10:07:39.687210 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.679915    2583 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 28 10:07:39.687210 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.679918    2583 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 28 10:07:39.687210 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.679921    2583 flags.go:64] FLAG: --fail-swap-on="true"
Jul 28 10:07:39.687210 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.679924    2583 flags.go:64] FLAG: --feature-gates=""
Jul 28 10:07:39.687210 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.679928    2583 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 28 10:07:39.687210 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.679931    2583 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 28 10:07:39.687210 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.679935    2583 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 28 10:07:39.687210 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.679938    2583 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 28 10:07:39.687210 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.679941    2583 flags.go:64] FLAG: --healthz-port="10248"
Jul 28 10:07:39.687210 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.679943    2583 flags.go:64] FLAG: --help="false"
Jul 28 10:07:39.687210 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.679946    2583 flags.go:64] FLAG: --hostname-override="ip-10-0-132-110.ec2.internal"
Jul 28 10:07:39.687210 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.679950    2583 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 28 10:07:39.687210 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680321    2583 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 28 10:07:39.687210 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680325    2583 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 28 10:07:39.687210 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680329    2583 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 28 10:07:39.687210 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680332    2583 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 28 10:07:39.687210 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680335    2583 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 28 10:07:39.687210 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680338    2583 flags.go:64] FLAG: --image-service-endpoint=""
Jul 28 10:07:39.687210 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680340    2583 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 28 10:07:39.687210 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680343    2583 flags.go:64] FLAG: --kube-api-burst="100"
Jul 28 10:07:39.687210 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680346    2583 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 28 10:07:39.687853 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680349    2583 flags.go:64] FLAG: --kube-api-qps="50"
Jul 28 10:07:39.687853 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680352    2583 flags.go:64] FLAG: --kube-reserved=""
Jul 28 10:07:39.687853 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680355    2583 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 28 10:07:39.687853 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680358    2583 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 28 10:07:39.687853 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680361    2583 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 28 10:07:39.687853 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680364    2583 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 28 10:07:39.687853 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680367    2583 flags.go:64] FLAG: --lock-file=""
Jul 28 10:07:39.687853 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680370    2583 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 28 10:07:39.687853 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680372    2583 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 28 10:07:39.687853 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680376    2583 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 28 10:07:39.687853 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680381    2583 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 28 10:07:39.687853 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680384    2583 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 28 10:07:39.687853 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680386    2583 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 28 10:07:39.687853 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680389    2583 flags.go:64] FLAG: --logging-format="text"
Jul 28 10:07:39.687853 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680392    2583 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 28 10:07:39.687853 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680395    2583 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 28 10:07:39.687853 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680398    2583 flags.go:64] FLAG: --manifest-url=""
Jul 28 10:07:39.687853 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680401    2583 flags.go:64] FLAG: --manifest-url-header=""
Jul 28 10:07:39.687853 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680405    2583 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 28 10:07:39.687853 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680408    2583 flags.go:64] FLAG: --max-open-files="1000000"
Jul 28 10:07:39.687853 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680412    2583 flags.go:64] FLAG: --max-pods="110"
Jul 28 10:07:39.687853 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680415    2583 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 28 10:07:39.687853 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680418    2583 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 28 10:07:39.687853 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680421    2583 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 28 10:07:39.687853 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680424    2583 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 28 10:07:39.688517 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680428    2583 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 28 10:07:39.688517 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680431    2583 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 28 10:07:39.688517 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680434    2583 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 28 10:07:39.688517 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680440    2583 flags.go:64] FLAG: --node-status-max-images="50"
Jul 28 10:07:39.688517 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680443    2583 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 28 10:07:39.688517 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680446    2583 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 28 10:07:39.688517 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680449    2583 flags.go:64] FLAG: --pod-cidr=""
Jul 28 10:07:39.688517 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680451    2583 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 28 10:07:39.688517 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680457    2583 flags.go:64] FLAG: --pod-manifest-path=""
Jul 28 10:07:39.688517 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680459    2583 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 28 10:07:39.688517 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680462    2583 flags.go:64] FLAG: --pods-per-core="0"
Jul 28 10:07:39.688517 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680466    2583 flags.go:64] FLAG: --port="10250"
Jul 28 10:07:39.688517 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680469    2583 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 28 10:07:39.688517 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680472    2583 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0239c3675a9339ff3"
Jul 28 10:07:39.688517 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680475    2583 flags.go:64] FLAG: --qos-reserved=""
Jul 28 10:07:39.688517 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680478    2583 flags.go:64] FLAG: --read-only-port="10255"
Jul 28 10:07:39.688517 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680481    2583 flags.go:64] FLAG: --register-node="true"
Jul 28 10:07:39.688517 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680483    2583 flags.go:64] FLAG: --register-with-taints=""
Jul 28 10:07:39.688517 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680487    2583 flags.go:64] FLAG: --registry-burst="10"
Jul 28 10:07:39.688517 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680489    2583 flags.go:64] FLAG: --registry-qps="5"
Jul 28 10:07:39.688517 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680492    2583 flags.go:64] FLAG: --reserved-cpus=""
Jul 28 10:07:39.688517 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680495    2583 flags.go:64] FLAG: --reserved-memory=""
Jul 28 10:07:39.688517 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680499    2583 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 28 10:07:39.688517 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680502    2583 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 28 10:07:39.688517 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680505    2583 flags.go:64] FLAG: --rotate-certificates="false"
Jul 28 10:07:39.689133 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680508    2583 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 28 10:07:39.689133 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680511    2583 flags.go:64] FLAG: --runonce="false"
Jul 28 10:07:39.689133 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680513    2583 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 28 10:07:39.689133 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680516    2583 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 28 10:07:39.689133 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680519    2583 flags.go:64] FLAG: --seccomp-default="false"
Jul 28 10:07:39.689133 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680522    2583 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 28 10:07:39.689133 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680525    2583 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 28 10:07:39.689133 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680529    2583 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 28 10:07:39.689133 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680532    2583 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 28 10:07:39.689133 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680535    2583 flags.go:64] FLAG: --storage-driver-password="root"
Jul 28 10:07:39.689133 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680537    2583 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 28 10:07:39.689133 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680540    2583 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 28 10:07:39.689133 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680543    2583 flags.go:64] FLAG: --storage-driver-user="root"
Jul 28 10:07:39.689133 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680545    2583 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 28 10:07:39.689133 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680548    2583 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 28 10:07:39.689133 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680551    2583 flags.go:64] FLAG: --system-cgroups=""
Jul 28 10:07:39.689133 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680554    2583 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 28 10:07:39.689133 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680559    2583 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 28 10:07:39.689133 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680561    2583 flags.go:64] FLAG: --tls-cert-file=""
Jul 28 10:07:39.689133 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680564    2583 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 28 10:07:39.689133 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680568    2583 flags.go:64] FLAG: --tls-min-version=""
Jul 28 10:07:39.689133 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680571    2583 flags.go:64] FLAG: --tls-private-key-file=""
Jul 28 10:07:39.689133 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680574    2583 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 28 10:07:39.689133 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680576    2583 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 28 10:07:39.689133 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680579    2583 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 28 10:07:39.689770 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680582    2583 flags.go:64] FLAG: --v="2"
Jul 28 10:07:39.689770 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680585    2583 flags.go:64] FLAG: --version="false"
Jul 28 10:07:39.689770 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680589    2583 flags.go:64] FLAG: --vmodule=""
Jul 28 10:07:39.689770 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680592    2583 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 28 10:07:39.689770 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.680595    2583 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 28 10:07:39.689770 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680701    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 10:07:39.689770 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680705    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 10:07:39.689770 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680708    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 10:07:39.689770 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680711    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 10:07:39.689770 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680714    2583 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 10:07:39.689770 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680717    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 10:07:39.689770 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680719    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 10:07:39.689770 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680722    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 10:07:39.689770 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680726    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 10:07:39.689770 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680729    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 10:07:39.689770 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680732    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 10:07:39.689770 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680734    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 10:07:39.689770 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680737    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 10:07:39.689770 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680739    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 10:07:39.689770 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680742    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 10:07:39.689770 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680744    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 10:07:39.690297 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680747    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 10:07:39.690297 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680750    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 10:07:39.690297 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680752    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 10:07:39.690297 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680755    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 10:07:39.690297 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680757    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 10:07:39.690297 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680759    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 10:07:39.690297 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680763    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 10:07:39.690297 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680766    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 10:07:39.690297 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680768    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 10:07:39.690297 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680771    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 10:07:39.690297 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680773    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 10:07:39.690297 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680776    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 10:07:39.690297 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680778    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 10:07:39.690297 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680781    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 10:07:39.690297 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680783    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 10:07:39.690297 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680787    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 10:07:39.690297 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680791    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 10:07:39.690297 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680793    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 10:07:39.690297 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680796    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 10:07:39.690772 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680799    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 10:07:39.690772 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680803    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 10:07:39.690772 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680807    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 10:07:39.690772 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680810    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 10:07:39.690772 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680812    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 10:07:39.690772 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680816    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 10:07:39.690772 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680819    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 10:07:39.690772 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680822    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 10:07:39.690772 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680825    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 10:07:39.690772 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680828    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 10:07:39.690772 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680831    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 10:07:39.690772 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680834    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 10:07:39.690772 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680836    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 10:07:39.690772 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680839    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 10:07:39.690772 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680842    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 10:07:39.690772 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680844    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 10:07:39.690772 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680847    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 10:07:39.690772 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680849    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 10:07:39.690772 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680852    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 10:07:39.691283 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680856    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 10:07:39.691283 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680858    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 10:07:39.691283 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680860    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 10:07:39.691283 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680863    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 10:07:39.691283 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680866    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 10:07:39.691283 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680868    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 10:07:39.691283 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680871    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 10:07:39.691283 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680875    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 10:07:39.691283 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680878    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 10:07:39.691283 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680881    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 10:07:39.691283 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680884    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 10:07:39.691283 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680887    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 10:07:39.691283 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680890    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 10:07:39.691283 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680892    2583 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 10:07:39.691283 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680895    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 10:07:39.691283 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680898    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 10:07:39.691283 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680900    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 10:07:39.691283 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680903    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 10:07:39.691283 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680907    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 10:07:39.691750 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680909    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 10:07:39.691750 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680912    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 10:07:39.691750 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680914    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 10:07:39.691750 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680917    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 10:07:39.691750 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680920    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 10:07:39.691750 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680923    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 10:07:39.691750 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680926    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 10:07:39.691750 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680929    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 10:07:39.691750 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680931    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 10:07:39.691750 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680934    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 10:07:39.691750 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680936    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 10:07:39.691750 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680939    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 10:07:39.691750 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680941    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 10:07:39.691750 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680945    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 10:07:39.691750 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680948    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 10:07:39.691750 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680950    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 10:07:39.691750 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680952    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 10:07:39.691750 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680955    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 10:07:39.691750 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680957    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 10:07:39.691750 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680960    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 10:07:39.692257 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680962    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 10:07:39.692257 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680965    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 10:07:39.692257 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680967    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 10:07:39.692257 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680970    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 10:07:39.692257 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680972    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 10:07:39.692257 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.680975    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 10:07:39.692257 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.681588    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 10:07:39.692257 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.688695    2583 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 28 10:07:39.692257 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.688712    2583 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 28 10:07:39.692257 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.688725    2583 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 28 10:07:39.692257 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.688731    2583 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 28 10:07:39.692257 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.688780    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 10:07:39.692257 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.688785    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 10:07:39.692257 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.688788    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 10:07:39.692619 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.688792    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 10:07:39.692619 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.688794    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 10:07:39.692619 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.688797    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 10:07:39.692619 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.688800    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 10:07:39.692619 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.688802    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 10:07:39.692619 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.688805    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 10:07:39.692619 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.688807    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 10:07:39.692619 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.688810    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 10:07:39.692619 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.688813    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 10:07:39.692619 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.688816    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 10:07:39.692619 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.688818    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 10:07:39.692619 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.688821    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 10:07:39.692619 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.688823    2583 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 10:07:39.692619 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.688826    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 10:07:39.692619 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.688829    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 10:07:39.692619 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.688831    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 10:07:39.692619 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.688834    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 10:07:39.692619 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.688836    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 10:07:39.692619 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.688839    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 10:07:39.692619 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.688841    2583 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 10:07:39.693099 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.688844    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 10:07:39.693099 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.688846    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 10:07:39.693099 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.688851    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 10:07:39.693099 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.688854    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 10:07:39.693099 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.688857    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 10:07:39.693099 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.688860    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 10:07:39.693099 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.688864    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 10:07:39.693099 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.688867    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 10:07:39.693099 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.688870    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 10:07:39.693099 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.688872    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 10:07:39.693099 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.688875    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 10:07:39.693099 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.688877    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 10:07:39.693099 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.688880    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 10:07:39.693099 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.688882    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 10:07:39.693099 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.688885    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 10:07:39.693099 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.688888    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 10:07:39.693099 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.688890    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 10:07:39.693099 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.688893    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 10:07:39.693099 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.688897    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 10:07:39.693584 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.688900    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 10:07:39.693584 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.688904    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 10:07:39.693584 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.688907    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 10:07:39.693584 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.688910    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 10:07:39.693584 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.688913    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 10:07:39.693584 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.688916    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 10:07:39.693584 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.688919    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 10:07:39.693584 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.688921    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 10:07:39.693584 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.688924    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 10:07:39.693584 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.688926    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 10:07:39.693584 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.688929    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 10:07:39.693584 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.688931    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 10:07:39.693584 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.688934    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 10:07:39.693584 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.688936    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 10:07:39.693584 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.688939    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 10:07:39.693584 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.688942    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 10:07:39.693584 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.688944    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 10:07:39.693584 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.688947    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 10:07:39.693584 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.688949    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 10:07:39.693584 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.688953    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 10:07:39.694109 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.688956    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 10:07:39.694109 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.688958    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 10:07:39.694109 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.688961    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 10:07:39.694109 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.688964    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 10:07:39.694109 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.688966    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 10:07:39.694109 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.688969    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 10:07:39.694109 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.688972    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 10:07:39.694109 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.688974    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 10:07:39.694109 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.688977    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 10:07:39.694109 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.688979    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 10:07:39.694109 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.688982    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 10:07:39.694109 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.688984    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 10:07:39.694109 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.688987    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 10:07:39.694109 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.688989    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 10:07:39.694109 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.688992    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 10:07:39.694109 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.688995    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 10:07:39.694109 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.688998    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 10:07:39.694109 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689001    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 10:07:39.694109 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689003    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 10:07:39.694586 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689006    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 10:07:39.694586 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689008    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 10:07:39.694586 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689011    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 10:07:39.694586 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689013    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 10:07:39.694586 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689016    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 10:07:39.694586 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689019    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 10:07:39.694586 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689021    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 10:07:39.694586 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689024    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 10:07:39.694586 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689027    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 10:07:39.694586 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689029    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 10:07:39.694586 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689032    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 10:07:39.694586 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689034    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 10:07:39.694586 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689037    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 10:07:39.694586 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689040    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 10:07:39.694586 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689042    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 10:07:39.694586 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689045    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 10:07:39.694586 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689047    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 10:07:39.694586 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689050    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 10:07:39.695033 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.689055    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 10:07:39.695033 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689190    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 10:07:39.695033 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689195    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 10:07:39.695033 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689198    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 10:07:39.695033 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689200    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 10:07:39.695033 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689203    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 10:07:39.695033 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689206    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 10:07:39.695033 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689209    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 10:07:39.695033 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689212    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 10:07:39.695033 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689215    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 10:07:39.695033 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689217    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 10:07:39.695033 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689220    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 10:07:39.695033 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689224    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 10:07:39.695033 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689226    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 10:07:39.695033 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689229    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 10:07:39.695440 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689231    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 10:07:39.695440 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689234    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 10:07:39.695440 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689236    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 10:07:39.695440 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689239    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 10:07:39.695440 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689242    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 10:07:39.695440 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689244    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 10:07:39.695440 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689247    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 10:07:39.695440 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689249    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 10:07:39.695440 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689252    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 10:07:39.695440 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689254    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 10:07:39.695440 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689257    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 10:07:39.695440 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689260    2583 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 10:07:39.695440 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689262    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 10:07:39.695440 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689265    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 10:07:39.695440 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689267    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 10:07:39.695440 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689270    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 10:07:39.695440 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689272    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 10:07:39.695440 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689275    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 10:07:39.695440 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689277    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 10:07:39.695440 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689280    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 10:07:39.695925 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689282    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 10:07:39.695925 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689285    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 10:07:39.695925 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689287    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 10:07:39.695925 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689290    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 10:07:39.695925 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689292    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 10:07:39.695925 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689295    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 10:07:39.695925 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689297    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 10:07:39.695925 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689300    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 10:07:39.695925 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689302    2583 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 10:07:39.695925 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689305    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 10:07:39.695925 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689308    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 10:07:39.695925 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689310    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 10:07:39.695925 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689313    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 10:07:39.695925 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689315    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 10:07:39.695925 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689318    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 10:07:39.695925 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689320    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 10:07:39.695925 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689323    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 10:07:39.695925 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689326    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 10:07:39.695925 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689328    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 10:07:39.696548 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689331    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 10:07:39.696548 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689333    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 10:07:39.696548 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689336    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 10:07:39.696548 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689338    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 10:07:39.696548 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689340    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 10:07:39.696548 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689343    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 10:07:39.696548 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689346    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 10:07:39.696548 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689348    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 10:07:39.696548 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689351    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 10:07:39.696548 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689353    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 10:07:39.696548 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689356    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 10:07:39.696548 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689358    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 10:07:39.696548 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689361    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 10:07:39.696548 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689364    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 10:07:39.696548 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689367    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 10:07:39.696548 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689369    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 10:07:39.696548 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689371    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 10:07:39.696548 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689374    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 10:07:39.696548 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689377    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 10:07:39.697005 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689379    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 10:07:39.697005 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689383    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 10:07:39.697005 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689386    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 10:07:39.697005 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689389    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 10:07:39.697005 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689392    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 10:07:39.697005 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689395    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 10:07:39.697005 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689398    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 10:07:39.697005 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689400    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 10:07:39.697005 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689402    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 10:07:39.697005 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689405    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 10:07:39.697005 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689408    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 10:07:39.697005 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689410    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 10:07:39.697005 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689413    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 10:07:39.697005 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689415    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 10:07:39.697005 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689417    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 10:07:39.697005 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689420    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 10:07:39.697005 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689422    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 10:07:39.697005 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689425    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 10:07:39.697005 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689427    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 10:07:39.697525 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689430    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 10:07:39.697525 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689433    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 10:07:39.697525 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689435    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 10:07:39.697525 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689437    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 10:07:39.697525 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689441    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 10:07:39.697525 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689445    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 10:07:39.697525 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689447    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 10:07:39.697525 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:39.689451    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 10:07:39.697525 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.689456    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 10:07:39.697525 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.689544    2583 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 28 10:07:39.697525 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.691469    2583 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 28 10:07:39.697525 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.692699    2583 server.go:1019] "Starting client certificate rotation"
Jul 28 10:07:39.697525 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.692800    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 10:07:39.697836 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.692834    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 10:07:39.697836 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.695351    2583 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 10:07:39.697836 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.696707    2583 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 10:07:39.710271 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.710256    2583 log.go:25] "Validated CRI v1 runtime API"
Jul 28 10:07:39.712696 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.712684    2583 log.go:25] "Validated CRI v1 image API"
Jul 28 10:07:39.713452 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.713437    2583 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 28 10:07:39.715537 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.715515    2583 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 983ddcab-3b27-4b49-b5bf-5b59e40a98c6:/dev/nvme0n1p3 d627b4c7-fb12-428c-9634-4a4f5192112b:/dev/nvme0n1p4]
Jul 28 10:07:39.715589 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.715537    2583 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 28 10:07:39.720596 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.720572    2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 10:07:39.722206 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.722073    2583 manager.go:216] Machine: {Timestamp:2026-07-28 10:07:39.720855962 +0000 UTC m=+0.311391777 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099409 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ee90121c0dd510d8766c5839e4d78 SystemUUID:ec2ee901-21c0-dd51-0d87-66c5839e4d78 BootID:f2570ee2-2a38-47ec-b914-703441fa6035 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:84:fe:11:a3:ad Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:84:fe:11:a3:ad Speed:0 Mtu:9001} {Name:ovs-system MacAddress:fa:43:70:c5:4e:76 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 28 10:07:39.722206 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.722204    2583 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 28 10:07:39.722321 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.722275    2583 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 28 10:07:39.725647 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.725622    2583 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 28 10:07:39.725774 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.725649    2583 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-110.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 28 10:07:39.726401 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.726391    2583 topology_manager.go:138] "Creating topology manager with none policy"
Jul 28 10:07:39.726432 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.726404    2583 container_manager_linux.go:309] "Creating device plugin manager"
Jul 28 10:07:39.726432 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.726411    2583 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 10:07:39.727143 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.727133    2583 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 10:07:39.728336 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.728326    2583 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 28 10:07:39.731022 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.731011    2583 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 10:07:39.731119 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.731110    2583 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 28 10:07:39.731183 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.731134    2583 kubelet.go:486] "Attempting to sync node with API server"
Jul 28 10:07:39.731183 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.731143    2583 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 28 10:07:39.731183 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.731173    2583 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 28 10:07:39.731183 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.731184    2583 kubelet.go:398] "Adding apiserver pod source"
Jul 28 10:07:39.731334 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.731193    2583 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 28 10:07:39.731544 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.731532    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 10:07:39.731574 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.731553    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 10:07:39.733320 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.733305    2583 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 28 10:07:39.734579 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.734567    2583 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 28 10:07:39.734631 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.734596    2583 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 28 10:07:39.735145 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.735134    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 28 10:07:39.735213 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.735200    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 28 10:07:39.735243 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.735217    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 28 10:07:39.735243 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.735224    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 28 10:07:39.735243 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.735233    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 28 10:07:39.735243 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.735238    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 28 10:07:39.735344 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.735245    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 28 10:07:39.735344 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.735251    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 28 10:07:39.735344 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.735257    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 28 10:07:39.735344 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.735262    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 28 10:07:39.735344 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.735272    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 28 10:07:39.735344 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.735277    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 28 10:07:39.736129 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.736115    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 28 10:07:39.736129 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.736126    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 28 10:07:39.739194 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.739180    2583 server.go:1268] "Started kubelet"
Jul 28 10:07:39.739314 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.739289    2583 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 28 10:07:39.739361 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.739280    2583 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 28 10:07:39.739361 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.739354    2583 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 28 10:07:39.739941 ip-10-0-132-110 systemd[1]: Started Kubernetes Kubelet.
Jul 28 10:07:39.740991 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.740973    2583 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 28 10:07:39.741643 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.741625    2583 server.go:310] "Adding debug handlers to kubelet server"
Jul 28 10:07:39.741808 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:39.741785    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-110.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 28 10:07:39.741885 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:39.741815    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 28 10:07:39.741987 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.741967    2583 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-110.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 28 10:07:39.743797 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:39.743779    2583 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 28 10:07:39.745029 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.745014    2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mt8ns"
Jul 28 10:07:39.747809 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:39.746953    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-110.ec2.internal.18c66c30eff6bdb7  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-110.ec2.internal,UID:ip-10-0-132-110.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-110.ec2.internal,},FirstTimestamp:2026-07-28 10:07:39.739135415 +0000 UTC m=+0.329671230,LastTimestamp:2026-07-28 10:07:39.739135415 +0000 UTC m=+0.329671230,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-110.ec2.internal,}"
Jul 28 10:07:39.748441 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.748427    2583 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 28 10:07:39.748506 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.748441    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 28 10:07:39.748566 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:39.748545    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-110.ec2.internal\" not found"
Jul 28 10:07:39.749017 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.749000    2583 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 28 10:07:39.749017 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.749007    2583 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 28 10:07:39.749160 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.749030    2583 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 28 10:07:39.749160 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.749119    2583 reconstruct.go:97] "Volume reconstruction finished"
Jul 28 10:07:39.749160 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.749129    2583 reconciler.go:29] "Reconciler: start to sync state"
Jul 28 10:07:39.750215 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.750193    2583 factory.go:55] Registering systemd factory
Jul 28 10:07:39.750301 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.750240    2583 factory.go:223] Registration of the systemd container factory successfully
Jul 28 10:07:39.750500 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.750485    2583 factory.go:153] Registering CRI-O factory
Jul 28 10:07:39.750500 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.750502    2583 factory.go:223] Registration of the crio container factory successfully
Jul 28 10:07:39.750619 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.750550    2583 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 28 10:07:39.750619 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.750575    2583 factory.go:103] Registering Raw factory
Jul 28 10:07:39.750619 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.750610    2583 manager.go:1196] Started watching for new ooms in manager
Jul 28 10:07:39.751601 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.751572    2583 manager.go:349] Starting recovery of all containers
Jul 28 10:07:39.752326 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:39.752300    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 28 10:07:39.752546 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:39.752516    2583 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-110.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 28 10:07:39.753641 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.753604    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mt8ns"
Jul 28 10:07:39.764271 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.764243    2583 manager.go:354] Recovery completed
Jul 28 10:07:39.768642 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.768628    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 10:07:39.770109 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.770095    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-110.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 10:07:39.770226 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.770130    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-110.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 10:07:39.770226 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.770140    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-110.ec2.internal" event="NodeHasSufficientPID"
Jul 28 10:07:39.770736 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.770709    2583 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 28 10:07:39.770787 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.770739    2583 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 28 10:07:39.770787 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.770757    2583 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 10:07:39.770853 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.770724    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 10:07:39.771556 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.771543    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-110.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 10:07:39.771621 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.771572    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-110.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 10:07:39.771621 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.771586    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-110.ec2.internal" event="NodeHasSufficientPID"
Jul 28 10:07:39.772081 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:39.771929    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-110.ec2.internal.18c66c30f1cf5a4f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-110.ec2.internal,UID:ip-10-0-132-110.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-110.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-110.ec2.internal,},FirstTimestamp:2026-07-28 10:07:39.770108495 +0000 UTC m=+0.360644309,LastTimestamp:2026-07-28 10:07:39.770108495 +0000 UTC m=+0.360644309,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-110.ec2.internal,}"
Jul 28 10:07:39.772250 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.772233    2583 policy_none.go:49] "None policy: Start"
Jul 28 10:07:39.772250 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.772252    2583 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 28 10:07:39.772418 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.772262    2583 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 28 10:07:39.773351 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.773339    2583 policy_none.go:47] "Start"
Jul 28 10:07:39.805690 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.805673    2583 manager.go:337] "Starting Device Plugin manager"
Jul 28 10:07:39.805764 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:39.805708    2583 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 28 10:07:39.805764 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.805718    2583 server.go:87] "Starting device plugin registration server"
Jul 28 10:07:39.805997 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.805984    2583 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 28 10:07:39.806044 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.806002    2583 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 28 10:07:39.806131 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.806114    2583 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 28 10:07:39.806528 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.806222    2583 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 28 10:07:39.806528 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.806241    2583 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 28 10:07:39.806638 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:39.806621    2583 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 28 10:07:39.806679 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:39.806662    2583 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-110.ec2.internal\" not found"
Jul 28 10:07:39.906351 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.906331    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 10:07:39.907499 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.907132    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-110.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 10:07:39.907499 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.907172    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-110.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 10:07:39.907499 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.907184    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-110.ec2.internal" event="NodeHasSufficientPID"
Jul 28 10:07:39.907499 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.907208    2583 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-110.ec2.internal"
Jul 28 10:07:39.915360 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.915342    2583 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-110.ec2.internal"
Jul 28 10:07:39.915431 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:39.915363    2583 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-110.ec2.internal\": node \"ip-10-0-132-110.ec2.internal\" not found"
Jul 28 10:07:39.932221 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:39.932174    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-110.ec2.internal\" not found"
Jul 28 10:07:39.940909 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.940890    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 28 10:07:39.942092 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.942078    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 28 10:07:39.942181 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.942136    2583 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 28 10:07:39.942181 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.942166    2583 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 28 10:07:39.942254 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:39.942201    2583 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 28 10:07:39.944120 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:39.944105    2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 10:07:40.033215 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:40.033195    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-110.ec2.internal\" not found"
Jul 28 10:07:40.042304 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:40.042286    2583 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-110.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-110.ec2.internal"]
Jul 28 10:07:40.042356 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:40.042341    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 10:07:40.043624 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:40.043611    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-110.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 10:07:40.043691 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:40.043636    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-110.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 10:07:40.043691 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:40.043655    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-110.ec2.internal" event="NodeHasSufficientPID"
Jul 28 10:07:40.045267 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:40.045254    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 10:07:40.045381 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:40.045368    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-110.ec2.internal"
Jul 28 10:07:40.045418 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:40.045395    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 10:07:40.045891 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:40.045877    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-110.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 10:07:40.045947 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:40.045891    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-110.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 10:07:40.045947 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:40.045906    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-110.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 10:07:40.045947 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:40.045923    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-110.ec2.internal" event="NodeHasSufficientPID"
Jul 28 10:07:40.046036 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:40.045909    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-110.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 10:07:40.046036 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:40.045986    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-110.ec2.internal" event="NodeHasSufficientPID"
Jul 28 10:07:40.047361 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:40.047348    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-110.ec2.internal"
Jul 28 10:07:40.047426 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:40.047377    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 10:07:40.047983 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:40.047968    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-110.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 10:07:40.048035 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:40.047997    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-110.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 10:07:40.048035 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:40.048010    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-110.ec2.internal" event="NodeHasSufficientPID"
Jul 28 10:07:40.051397 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:40.051291    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2b84527c672e7fff4b21007a2e0dc2c6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-110.ec2.internal\" (UID: \"2b84527c672e7fff4b21007a2e0dc2c6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-110.ec2.internal"
Jul 28 10:07:40.051397 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:40.051317    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b84527c672e7fff4b21007a2e0dc2c6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-110.ec2.internal\" (UID: \"2b84527c672e7fff4b21007a2e0dc2c6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-110.ec2.internal"
Jul 28 10:07:40.051397 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:40.051334    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c2a2b445e0df91dcaba3ba620484e779-config\") pod \"kube-apiserver-proxy-ip-10-0-132-110.ec2.internal\" (UID: \"c2a2b445e0df91dcaba3ba620484e779\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-110.ec2.internal"
Jul 28 10:07:41.557891 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:41.557870    2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-110.ec2.internal\" not found" node="ip-10-0-132-110.ec2.internal"
Jul 28 10:07:41.562758 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:41.562740    2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-110.ec2.internal\" not found" node="ip-10-0-132-110.ec2.internal"
Jul 28 10:07:41.624244 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:41.624223    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-110.ec2.internal\" not found"
Jul 28 10:07:41.642427 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:41.642405    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2b84527c672e7fff4b21007a2e0dc2c6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-110.ec2.internal\" (UID: \"2b84527c672e7fff4b21007a2e0dc2c6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-110.ec2.internal"
Jul 28 10:07:41.642482 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:41.642432    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b84527c672e7fff4b21007a2e0dc2c6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-110.ec2.internal\" (UID: \"2b84527c672e7fff4b21007a2e0dc2c6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-110.ec2.internal"
Jul 28 10:07:41.642482 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:41.642448    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c2a2b445e0df91dcaba3ba620484e779-config\") pod \"kube-apiserver-proxy-ip-10-0-132-110.ec2.internal\" (UID: \"c2a2b445e0df91dcaba3ba620484e779\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-110.ec2.internal"
Jul 28 10:07:41.642544 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:41.642497    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c2a2b445e0df91dcaba3ba620484e779-config\") pod \"kube-apiserver-proxy-ip-10-0-132-110.ec2.internal\" (UID: \"c2a2b445e0df91dcaba3ba620484e779\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-110.ec2.internal"
Jul 28 10:07:41.642544 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:41.642501    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2b84527c672e7fff4b21007a2e0dc2c6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-110.ec2.internal\" (UID: \"2b84527c672e7fff4b21007a2e0dc2c6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-110.ec2.internal"
Jul 28 10:07:41.642544 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:41.642522    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b84527c672e7fff4b21007a2e0dc2c6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-110.ec2.internal\" (UID: \"2b84527c672e7fff4b21007a2e0dc2c6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-110.ec2.internal"
Jul 28 10:07:41.724328 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:41.724272    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-110.ec2.internal\" not found"
Jul 28 10:07:41.824679 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:41.824654    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-110.ec2.internal\" not found"
Jul 28 10:07:41.860152 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:41.860136    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-110.ec2.internal"
Jul 28 10:07:41.866269 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:41.866251    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-110.ec2.internal"
Jul 28 10:07:41.925327 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:41.925309    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-110.ec2.internal\" not found"
Jul 28 10:07:42.025793 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:42.025744    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-110.ec2.internal\" not found"
Jul 28 10:07:42.126277 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:42.126252    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-110.ec2.internal\" not found"
Jul 28 10:07:42.182678 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:42.182660    2583 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 28 10:07:42.183035 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:42.182782    2583 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 10:07:42.227090 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:42.227064    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-110.ec2.internal\" not found"
Jul 28 10:07:42.248386 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:42.248359    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 10:02:41 +0000 UTC" deadline="2028-03-01 23:32:23.716962303 +0000 UTC"
Jul 28 10:07:42.248386 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:42.248381    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13981h24m41.468583374s"
Jul 28 10:07:42.327699 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:42.327655    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-110.ec2.internal\" not found"
Jul 28 10:07:42.428791 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:42.428762    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-110.ec2.internal\" not found"
Jul 28 10:07:42.429463 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:42.429447    2583 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 10:07:42.441769 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:42.441730    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b84527c672e7fff4b21007a2e0dc2c6.slice/crio-6266dafceb756cec826760f6c07786cab08c997a20c7296a6795029d778a6c06 WatchSource:0}: Error finding container 6266dafceb756cec826760f6c07786cab08c997a20c7296a6795029d778a6c06: Status 404 returned error can't find the container with id 6266dafceb756cec826760f6c07786cab08c997a20c7296a6795029d778a6c06
Jul 28 10:07:42.445493 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:42.445480    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 10:07:42.529206 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:42.529170    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-110.ec2.internal\" not found"
Jul 28 10:07:42.530058 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:42.530043    2583 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 10:07:42.582668 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:42.582638    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2a2b445e0df91dcaba3ba620484e779.slice/crio-6d5eb2b15c798724048d50ec43baff6ec4c514380ea8eada9a4b1962777c1319 WatchSource:0}: Error finding container 6d5eb2b15c798724048d50ec43baff6ec4c514380ea8eada9a4b1962777c1319: Status 404 returned error can't find the container with id 6d5eb2b15c798724048d50ec43baff6ec4c514380ea8eada9a4b1962777c1319
Jul 28 10:07:42.630228 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:42.630209    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-110.ec2.internal\" not found"
Jul 28 10:07:42.667880 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:42.667866    2583 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 10:07:42.738819 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:42.738804    2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-110.ec2.internal"
Jul 28 10:07:42.750592 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:42.750576    2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 10:07:42.751631 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:42.751618    2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-110.ec2.internal"
Jul 28 10:07:42.760524 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:42.760508    2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 10:07:43.222730 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.222702    2583 apiserver.go:52] "Watching apiserver"
Jul 28 10:07:43.229165 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.229139    2583 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 28 10:07:43.229630 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.229601    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-132-110.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vrkpg","openshift-cluster-node-tuning-operator/tuned-vvkrx","openshift-image-registry/node-ca-v94ww","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-110.ec2.internal","openshift-multus/network-metrics-daemon-ldq5b","openshift-network-operator/iptables-alerter-5z6x8","kube-system/global-pull-secret-syncer-64pm8","kube-system/konnectivity-agent-lqnk6","openshift-dns/node-resolver-nxpwb","openshift-multus/multus-additional-cni-plugins-545mm","openshift-multus/multus-jlf2c","openshift-network-diagnostics/network-check-target-7dfzm","openshift-ovn-kubernetes/ovnkube-node-rrxnl"]
Jul 28 10:07:43.231270 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.231250    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5z6x8"
Jul 28 10:07:43.233152 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.232611    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-545mm"
Jul 28 10:07:43.233949 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.233538    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 28 10:07:43.233949 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.233616    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-prmcj\""
Jul 28 10:07:43.233949 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.233544    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 28 10:07:43.233949 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.233821    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 28 10:07:43.234196 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.233988    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vvkrx"
Jul 28 10:07:43.236209 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.234644    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 28 10:07:43.236209 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.234984    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 28 10:07:43.236209 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.235198    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 28 10:07:43.236209 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.235539    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 28 10:07:43.236209 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.235754    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 28 10:07:43.236209 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.235937    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-8m7kv\""
Jul 28 10:07:43.236209 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.236145    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-v94ww"
Jul 28 10:07:43.236576 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.236403    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 28 10:07:43.236628 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.236603    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-mnznw\""
Jul 28 10:07:43.237045 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.236700    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 28 10:07:43.237229 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.237212    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rrxnl"
Jul 28 10:07:43.241199 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.239125    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 28 10:07:43.241199 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.239756    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 28 10:07:43.241199 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.240071    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldq5b"
Jul 28 10:07:43.241199 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:43.240358    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldq5b" podUID="19db4187-c2f2-46d9-bb21-ce1d3c12c422"
Jul 28 10:07:43.241199 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.240755    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 28 10:07:43.241199 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.241087    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hl8q4\""
Jul 28 10:07:43.241656 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.241333    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 28 10:07:43.241656 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.241411    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 28 10:07:43.241656 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.241582    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 28 10:07:43.241802 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.241669    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 28 10:07:43.241852 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.241824    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 28 10:07:43.242107 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.242036    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vrs9p\""
Jul 28 10:07:43.243598 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.242384    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 28 10:07:43.243598 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.242724    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lqnk6"
Jul 28 10:07:43.243598 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.243546    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 28 10:07:43.244231 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.244212    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7dfzm"
Jul 28 10:07:43.244329 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:43.244287    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7dfzm" podUID="d329552c-d2a3-4960-9130-bc9e211efc18"
Jul 28 10:07:43.245619 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.245258    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 28 10:07:43.245619 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.245374    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-g9hk5\""
Jul 28 10:07:43.245619 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.245616    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 28 10:07:43.246767 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.246746    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vrkpg"
Jul 28 10:07:43.249823 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.249804    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 28 10:07:43.249910 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.249866    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-pwk2p\""
Jul 28 10:07:43.250829 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.250042    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 28 10:07:43.250829 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.250099    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 28 10:07:43.250829 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.250280    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nxpwb"
Jul 28 10:07:43.251492 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.251477    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-64pm8"
Jul 28 10:07:43.251661 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:43.251641    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-64pm8" podUID="07ad2626-9f01-48ec-adb7-c5f4ab340b7a"
Jul 28 10:07:43.251823 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.251808    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8490536f-2f40-4389-bf3f-d5cab1ab3a00-konnectivity-ca\") pod \"konnectivity-agent-lqnk6\" (UID: \"8490536f-2f40-4389-bf3f-d5cab1ab3a00\") " pod="kube-system/konnectivity-agent-lqnk6"
Jul 28 10:07:43.251931 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.251915    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/675b6659-b757-4eca-ba86-28b21af90501-system-cni-dir\") pod \"multus-additional-cni-plugins-545mm\" (UID: \"675b6659-b757-4eca-ba86-28b21af90501\") " pod="openshift-multus/multus-additional-cni-plugins-545mm"
Jul 28 10:07:43.252018 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.252006    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ca02c39d-85bd-4cea-ab02-19f67555f81a-run-ovn\") pod \"ovnkube-node-rrxnl\" (UID: \"ca02c39d-85bd-4cea-ab02-19f67555f81a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrxnl"
Jul 28 10:07:43.252116 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.252100    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ca02c39d-85bd-4cea-ab02-19f67555f81a-log-socket\") pod \"ovnkube-node-rrxnl\" (UID: \"ca02c39d-85bd-4cea-ab02-19f67555f81a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrxnl"
Jul 28 10:07:43.252221 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.252207    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ca02c39d-85bd-4cea-ab02-19f67555f81a-env-overrides\") pod \"ovnkube-node-rrxnl\" (UID: \"ca02c39d-85bd-4cea-ab02-19f67555f81a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrxnl"
Jul 28 10:07:43.252322 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.252309    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ca02c39d-85bd-4cea-ab02-19f67555f81a-ovnkube-script-lib\") pod \"ovnkube-node-rrxnl\" (UID: \"ca02c39d-85bd-4cea-ab02-19f67555f81a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrxnl"
Jul 28 10:07:43.252414 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.252400    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/17fe592b-e2cc-42ad-8803-0944d20e4dc0-etc-sysconfig\") pod \"tuned-vvkrx\" (UID: \"17fe592b-e2cc-42ad-8803-0944d20e4dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-vvkrx"
Jul 28 10:07:43.252530 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.252493    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/17fe592b-e2cc-42ad-8803-0944d20e4dc0-etc-kubernetes\") pod \"tuned-vvkrx\" (UID: \"17fe592b-e2cc-42ad-8803-0944d20e4dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-vvkrx"
Jul 28 10:07:43.252879 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.252825    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 28 10:07:43.252950 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.252878    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 28 10:07:43.253006 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.252835    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-psq46\""
Jul 28 10:07:43.253006 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.252861    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/17fe592b-e2cc-42ad-8803-0944d20e4dc0-host\") pod \"tuned-vvkrx\" (UID: \"17fe592b-e2cc-42ad-8803-0944d20e4dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-vvkrx"
Jul 28 10:07:43.253103 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.253034    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbvk7\" (UniqueName: \"kubernetes.io/projected/17fe592b-e2cc-42ad-8803-0944d20e4dc0-kube-api-access-jbvk7\") pod \"tuned-vvkrx\" (UID: \"17fe592b-e2cc-42ad-8803-0944d20e4dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-vvkrx"
Jul 28 10:07:43.253103 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.253061    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gzlj\" (UniqueName: \"kubernetes.io/projected/d329552c-d2a3-4960-9130-bc9e211efc18-kube-api-access-2gzlj\") pod \"network-check-target-7dfzm\" (UID: \"d329552c-d2a3-4960-9130-bc9e211efc18\") " pod="openshift-network-diagnostics/network-check-target-7dfzm"
Jul 28 10:07:43.253103 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.253092    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ca02c39d-85bd-4cea-ab02-19f67555f81a-node-log\") pod \"ovnkube-node-rrxnl\" (UID: \"ca02c39d-85bd-4cea-ab02-19f67555f81a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrxnl"
Jul 28 10:07:43.253326 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.253118    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1a872d0f-af6f-4fd4-bf8f-f763ce610da4-iptables-alerter-script\") pod \"iptables-alerter-5z6x8\" (UID: \"1a872d0f-af6f-4fd4-bf8f-f763ce610da4\") " pod="openshift-network-operator/iptables-alerter-5z6x8"
Jul 28 10:07:43.253326 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.253148    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/17fe592b-e2cc-42ad-8803-0944d20e4dc0-etc-tuned\") pod \"tuned-vvkrx\" (UID: \"17fe592b-e2cc-42ad-8803-0944d20e4dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-vvkrx"
Jul 28 10:07:43.253326 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.253173    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/675b6659-b757-4eca-ba86-28b21af90501-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-545mm\" (UID: \"675b6659-b757-4eca-ba86-28b21af90501\") " pod="openshift-multus/multus-additional-cni-plugins-545mm"
Jul 28 10:07:43.253326 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.253299    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca02c39d-85bd-4cea-ab02-19f67555f81a-host-run-ovn-kubernetes\") pod \"ovnkube-node-rrxnl\" (UID: \"ca02c39d-85bd-4cea-ab02-19f67555f81a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrxnl"
Jul 28 10:07:43.253326 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.253322    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ca02c39d-85bd-4cea-ab02-19f67555f81a-host-cni-bin\") pod \"ovnkube-node-rrxnl\" (UID: \"ca02c39d-85bd-4cea-ab02-19f67555f81a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrxnl"
Jul 28 10:07:43.253619 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.253391    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca02c39d-85bd-4cea-ab02-19f67555f81a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rrxnl\" (UID: \"ca02c39d-85bd-4cea-ab02-19f67555f81a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrxnl"
Jul 28 10:07:43.253619 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.253458    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1a872d0f-af6f-4fd4-bf8f-f763ce610da4-host-slash\") pod \"iptables-alerter-5z6x8\" (UID: \"1a872d0f-af6f-4fd4-bf8f-f763ce610da4\") " pod="openshift-network-operator/iptables-alerter-5z6x8"
Jul 28 10:07:43.253619 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.253485    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/675b6659-b757-4eca-ba86-28b21af90501-cnibin\") pod \"multus-additional-cni-plugins-545mm\" (UID: \"675b6659-b757-4eca-ba86-28b21af90501\") " pod="openshift-multus/multus-additional-cni-plugins-545mm"
Jul 28 10:07:43.253759 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.253636    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ca02c39d-85bd-4cea-ab02-19f67555f81a-host-run-netns\") pod \"ovnkube-node-rrxnl\" (UID: \"ca02c39d-85bd-4cea-ab02-19f67555f81a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrxnl"
Jul 28 10:07:43.253759 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.253663    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca02c39d-85bd-4cea-ab02-19f67555f81a-etc-openvswitch\") pod \"ovnkube-node-rrxnl\" (UID: \"ca02c39d-85bd-4cea-ab02-19f67555f81a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrxnl"
Jul 28 10:07:43.253759 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.253685    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/17fe592b-e2cc-42ad-8803-0944d20e4dc0-run\") pod \"tuned-vvkrx\" (UID: \"17fe592b-e2cc-42ad-8803-0944d20e4dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-vvkrx"
Jul 28 10:07:43.253759 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.253707    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19db4187-c2f2-46d9-bb21-ce1d3c12c422-metrics-certs\") pod \"network-metrics-daemon-ldq5b\" (UID: \"19db4187-c2f2-46d9-bb21-ce1d3c12c422\") " pod="openshift-multus/network-metrics-daemon-ldq5b"
Jul 28 10:07:43.253759 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.253729    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8490536f-2f40-4389-bf3f-d5cab1ab3a00-agent-certs\") pod \"konnectivity-agent-lqnk6\" (UID: \"8490536f-2f40-4389-bf3f-d5cab1ab3a00\") " pod="kube-system/konnectivity-agent-lqnk6"
Jul 28 10:07:43.254053 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.253930    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jlf2c"
Jul 28 10:07:43.254216 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.254195    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/675b6659-b757-4eca-ba86-28b21af90501-os-release\") pod \"multus-additional-cni-plugins-545mm\" (UID: \"675b6659-b757-4eca-ba86-28b21af90501\") " pod="openshift-multus/multus-additional-cni-plugins-545mm"
Jul 28 10:07:43.254288 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.254248    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/675b6659-b757-4eca-ba86-28b21af90501-cni-binary-copy\") pod \"multus-additional-cni-plugins-545mm\" (UID: \"675b6659-b757-4eca-ba86-28b21af90501\") " pod="openshift-multus/multus-additional-cni-plugins-545mm"
Jul 28 10:07:43.254340 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.254287    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca02c39d-85bd-4cea-ab02-19f67555f81a-host-kubelet\") pod \"ovnkube-node-rrxnl\" (UID: \"ca02c39d-85bd-4cea-ab02-19f67555f81a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrxnl"
Jul 28 10:07:43.254340 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.254314    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ca02c39d-85bd-4cea-ab02-19f67555f81a-systemd-units\") pod \"ovnkube-node-rrxnl\" (UID: \"ca02c39d-85bd-4cea-ab02-19f67555f81a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrxnl"
Jul 28 10:07:43.254340 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.254335    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ca02c39d-85bd-4cea-ab02-19f67555f81a-ovn-node-metrics-cert\") pod \"ovnkube-node-rrxnl\" (UID: \"ca02c39d-85bd-4cea-ab02-19f67555f81a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrxnl"
Jul 28 10:07:43.254482 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.254384    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/17fe592b-e2cc-42ad-8803-0944d20e4dc0-etc-modprobe-d\") pod \"tuned-vvkrx\" (UID: \"17fe592b-e2cc-42ad-8803-0944d20e4dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-vvkrx"
Jul 28 10:07:43.254482 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.254414    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/17fe592b-e2cc-42ad-8803-0944d20e4dc0-etc-sysctl-conf\") pod \"tuned-vvkrx\" (UID: \"17fe592b-e2cc-42ad-8803-0944d20e4dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-vvkrx"
Jul 28 10:07:43.254482 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.254441    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsskl\" (UniqueName: \"kubernetes.io/projected/19db4187-c2f2-46d9-bb21-ce1d3c12c422-kube-api-access-vsskl\") pod \"network-metrics-daemon-ldq5b\" (UID: \"19db4187-c2f2-46d9-bb21-ce1d3c12c422\") " pod="openshift-multus/network-metrics-daemon-ldq5b"
Jul 28 10:07:43.254482 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.254466    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/675b6659-b757-4eca-ba86-28b21af90501-tuning-conf-dir\") pod \"multus-additional-cni-plugins-545mm\" (UID: \"675b6659-b757-4eca-ba86-28b21af90501\") " pod="openshift-multus/multus-additional-cni-plugins-545mm"
Jul 28 10:07:43.254672 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.254490    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ca02c39d-85bd-4cea-ab02-19f67555f81a-host-cni-netd\") pod \"ovnkube-node-rrxnl\" (UID: \"ca02c39d-85bd-4cea-ab02-19f67555f81a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrxnl"
Jul 28 10:07:43.254672 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.254513    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4jr6\" (UniqueName: \"kubernetes.io/projected/ca02c39d-85bd-4cea-ab02-19f67555f81a-kube-api-access-b4jr6\") pod \"ovnkube-node-rrxnl\" (UID: \"ca02c39d-85bd-4cea-ab02-19f67555f81a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrxnl"
Jul 28 10:07:43.254672 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.254537    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl6x8\" (UniqueName: \"kubernetes.io/projected/1a872d0f-af6f-4fd4-bf8f-f763ce610da4-kube-api-access-jl6x8\") pod \"iptables-alerter-5z6x8\" (UID: \"1a872d0f-af6f-4fd4-bf8f-f763ce610da4\") " pod="openshift-network-operator/iptables-alerter-5z6x8"
Jul 28 10:07:43.254672 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.254558    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/17fe592b-e2cc-42ad-8803-0944d20e4dc0-etc-sysctl-d\") pod \"tuned-vvkrx\" (UID: \"17fe592b-e2cc-42ad-8803-0944d20e4dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-vvkrx"
Jul 28 10:07:43.254672 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.254580    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/17fe592b-e2cc-42ad-8803-0944d20e4dc0-lib-modules\") pod \"tuned-vvkrx\" (UID: \"17fe592b-e2cc-42ad-8803-0944d20e4dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-vvkrx"
Jul 28 10:07:43.254672 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.254601    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17fe592b-e2cc-42ad-8803-0944d20e4dc0-tmp\") pod \"tuned-vvkrx\" (UID: \"17fe592b-e2cc-42ad-8803-0944d20e4dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-vvkrx"
Jul 28 10:07:43.254672 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.254633    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw9j2\" (UniqueName: \"kubernetes.io/projected/675b6659-b757-4eca-ba86-28b21af90501-kube-api-access-hw9j2\") pod \"multus-additional-cni-plugins-545mm\" (UID: \"675b6659-b757-4eca-ba86-28b21af90501\") " pod="openshift-multus/multus-additional-cni-plugins-545mm"
Jul 28 10:07:43.254672 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.254656    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ca02c39d-85bd-4cea-ab02-19f67555f81a-run-systemd\") pod \"ovnkube-node-rrxnl\" (UID: \"ca02c39d-85bd-4cea-ab02-19f67555f81a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrxnl"
Jul 28 10:07:43.255031 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.254677    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca02c39d-85bd-4cea-ab02-19f67555f81a-run-openvswitch\") pod \"ovnkube-node-rrxnl\" (UID: \"ca02c39d-85bd-4cea-ab02-19f67555f81a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrxnl"
Jul 28 10:07:43.255031 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.254699    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ca02c39d-85bd-4cea-ab02-19f67555f81a-ovnkube-config\") pod \"ovnkube-node-rrxnl\" (UID: \"ca02c39d-85bd-4cea-ab02-19f67555f81a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrxnl"
Jul 28 10:07:43.255031 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.254723    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/17fe592b-e2cc-42ad-8803-0944d20e4dc0-etc-systemd\") pod \"tuned-vvkrx\" (UID: \"17fe592b-e2cc-42ad-8803-0944d20e4dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-vvkrx"
Jul 28 10:07:43.255031 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.254744    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/17fe592b-e2cc-42ad-8803-0944d20e4dc0-sys\") pod \"tuned-vvkrx\" (UID: \"17fe592b-e2cc-42ad-8803-0944d20e4dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-vvkrx"
Jul 28 10:07:43.255031 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.254782    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/17fe592b-e2cc-42ad-8803-0944d20e4dc0-var-lib-kubelet\") pod \"tuned-vvkrx\" (UID: \"17fe592b-e2cc-42ad-8803-0944d20e4dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-vvkrx"
Jul 28 10:07:43.255031 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.254835    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eeccf40b-0096-40d3-b231-d7ce00688cc9-host\") pod \"node-ca-v94ww\" (UID: \"eeccf40b-0096-40d3-b231-d7ce00688cc9\") " pod="openshift-image-registry/node-ca-v94ww"
Jul 28 10:07:43.255031 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.254859    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/675b6659-b757-4eca-ba86-28b21af90501-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-545mm\" (UID: \"675b6659-b757-4eca-ba86-28b21af90501\") " pod="openshift-multus/multus-additional-cni-plugins-545mm"
Jul 28 10:07:43.255031 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.254882    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ca02c39d-85bd-4cea-ab02-19f67555f81a-host-slash\") pod \"ovnkube-node-rrxnl\" (UID: \"ca02c39d-85bd-4cea-ab02-19f67555f81a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrxnl"
Jul 28 10:07:43.255031 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.254903    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca02c39d-85bd-4cea-ab02-19f67555f81a-var-lib-openvswitch\") pod \"ovnkube-node-rrxnl\" (UID: \"ca02c39d-85bd-4cea-ab02-19f67555f81a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrxnl"
Jul 28 10:07:43.255031 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.254924    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/eeccf40b-0096-40d3-b231-d7ce00688cc9-serviceca\") pod \"node-ca-v94ww\" (UID: \"eeccf40b-0096-40d3-b231-d7ce00688cc9\") " pod="openshift-image-registry/node-ca-v94ww"
Jul 28 10:07:43.255031 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.254946    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fzw8\" (UniqueName: \"kubernetes.io/projected/eeccf40b-0096-40d3-b231-d7ce00688cc9-kube-api-access-2fzw8\") pod \"node-ca-v94ww\" (UID: \"eeccf40b-0096-40d3-b231-d7ce00688cc9\") " pod="openshift-image-registry/node-ca-v94ww"
Jul 28 10:07:43.256243 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.255598    2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 10:07:43.256243 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.256065    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 28 10:07:43.257245 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.257020    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-w7679\""
Jul 28 10:07:43.280011 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.279988    2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-4dpnh"
Jul 28 10:07:43.288017 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.287980    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-4dpnh"
Jul 28 10:07:43.340812 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.340787    2583 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 28 10:07:43.355700 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.355673    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/17fe592b-e2cc-42ad-8803-0944d20e4dc0-etc-systemd\") pod \"tuned-vvkrx\" (UID: \"17fe592b-e2cc-42ad-8803-0944d20e4dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-vvkrx"
Jul 28 10:07:43.355814 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.355712    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eeccf40b-0096-40d3-b231-d7ce00688cc9-host\") pod \"node-ca-v94ww\" (UID: \"eeccf40b-0096-40d3-b231-d7ce00688cc9\") " pod="openshift-image-registry/node-ca-v94ww"
Jul 28 10:07:43.355814 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.355737    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/675b6659-b757-4eca-ba86-28b21af90501-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-545mm\" (UID: \"675b6659-b757-4eca-ba86-28b21af90501\") " pod="openshift-multus/multus-additional-cni-plugins-545mm"
Jul 28 10:07:43.355814 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.355762    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca02c39d-85bd-4cea-ab02-19f67555f81a-var-lib-openvswitch\") pod \"ovnkube-node-rrxnl\" (UID: \"ca02c39d-85bd-4cea-ab02-19f67555f81a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrxnl"
Jul 28 10:07:43.355814 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.355786    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ca02c39d-85bd-4cea-ab02-19f67555f81a-run-ovn\") pod \"ovnkube-node-rrxnl\" (UID: \"ca02c39d-85bd-4cea-ab02-19f67555f81a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrxnl"
Jul 28 10:07:43.356042 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.355846    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca02c39d-85bd-4cea-ab02-19f67555f81a-var-lib-openvswitch\") pod \"ovnkube-node-rrxnl\" (UID: \"ca02c39d-85bd-4cea-ab02-19f67555f81a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrxnl"
Jul 28 10:07:43.356042 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.355851    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eeccf40b-0096-40d3-b231-d7ce00688cc9-host\") pod \"node-ca-v94ww\" (UID: \"eeccf40b-0096-40d3-b231-d7ce00688cc9\") " pod="openshift-image-registry/node-ca-v94ww"
Jul 28 10:07:43.356042 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.355865    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/17fe592b-e2cc-42ad-8803-0944d20e4dc0-etc-systemd\") pod \"tuned-vvkrx\" (UID: \"17fe592b-e2cc-42ad-8803-0944d20e4dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-vvkrx"
Jul 28 10:07:43.356042 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.355891    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ab7f977a-7d91-4d97-85ca-f07a9d373ee8-host-var-lib-cni-bin\") pod \"multus-jlf2c\" (UID: \"ab7f977a-7d91-4d97-85ca-f07a9d373ee8\") " pod="openshift-multus/multus-jlf2c"
Jul 28 10:07:43.356042 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.355910    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ca02c39d-85bd-4cea-ab02-19f67555f81a-run-ovn\") pod \"ovnkube-node-rrxnl\" (UID: \"ca02c39d-85bd-4cea-ab02-19f67555f81a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrxnl"
Jul 28 10:07:43.356042 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.355913    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8490536f-2f40-4389-bf3f-d5cab1ab3a00-konnectivity-ca\") pod \"konnectivity-agent-lqnk6\" (UID: \"8490536f-2f40-4389-bf3f-d5cab1ab3a00\") " pod="kube-system/konnectivity-agent-lqnk6"
Jul 28 10:07:43.356042 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.355939    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42229f5e-156f-4abd-a2ee-700c9f340775-tmp-dir\") pod \"node-resolver-nxpwb\" (UID: \"42229f5e-156f-4abd-a2ee-700c9f340775\") " pod="openshift-dns/node-resolver-nxpwb"
Jul 28 10:07:43.356042 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.355975    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd8b4\" (UniqueName: \"kubernetes.io/projected/55104d0d-d660-4142-9b73-3277e628eefa-kube-api-access-kd8b4\") pod \"aws-ebs-csi-driver-node-vrkpg\" (UID: \"55104d0d-d660-4142-9b73-3277e628eefa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vrkpg"
Jul 28 10:07:43.356042 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.356003    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/17fe592b-e2cc-42ad-8803-0944d20e4dc0-host\") pod \"tuned-vvkrx\" (UID: \"17fe592b-e2cc-42ad-8803-0944d20e4dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-vvkrx"
Jul 28 10:07:43.356042 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.356036    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbvk7\" (UniqueName: \"kubernetes.io/projected/17fe592b-e2cc-42ad-8803-0944d20e4dc0-kube-api-access-jbvk7\") pod \"tuned-vvkrx\" (UID: \"17fe592b-e2cc-42ad-8803-0944d20e4dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-vvkrx"
Jul 28 10:07:43.356548 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.356065    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/17fe592b-e2cc-42ad-8803-0944d20e4dc0-host\") pod \"tuned-vvkrx\" (UID: \"17fe592b-e2cc-42ad-8803-0944d20e4dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-vvkrx"
Jul 28 10:07:43.356548 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.356190    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gzlj\" (UniqueName: \"kubernetes.io/projected/d329552c-d2a3-4960-9130-bc9e211efc18-kube-api-access-2gzlj\") pod \"network-check-target-7dfzm\" (UID: \"d329552c-d2a3-4960-9130-bc9e211efc18\") " pod="openshift-network-diagnostics/network-check-target-7dfzm"
Jul 28 10:07:43.356548 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.356301    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ca02c39d-85bd-4cea-ab02-19f67555f81a-node-log\") pod \"ovnkube-node-rrxnl\" (UID: \"ca02c39d-85bd-4cea-ab02-19f67555f81a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrxnl"
Jul 28 10:07:43.356548 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.356328    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca02c39d-85bd-4cea-ab02-19f67555f81a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rrxnl\" (UID: \"ca02c39d-85bd-4cea-ab02-19f67555f81a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrxnl"
Jul 28 10:07:43.356548 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.356375    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/42229f5e-156f-4abd-a2ee-700c9f340775-hosts-file\") pod \"node-resolver-nxpwb\" (UID: \"42229f5e-156f-4abd-a2ee-700c9f340775\") " pod="openshift-dns/node-resolver-nxpwb"
Jul 28 10:07:43.356548 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.356398    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/55104d0d-d660-4142-9b73-3277e628eefa-registration-dir\") pod \"aws-ebs-csi-driver-node-vrkpg\" (UID: \"55104d0d-d660-4142-9b73-3277e628eefa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vrkpg"
Jul 28 10:07:43.356548 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.356444    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ca02c39d-85bd-4cea-ab02-19f67555f81a-node-log\") pod \"ovnkube-node-rrxnl\" (UID: \"ca02c39d-85bd-4cea-ab02-19f67555f81a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrxnl"
Jul 28 10:07:43.356548 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.356498    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca02c39d-85bd-4cea-ab02-19f67555f81a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rrxnl\" (UID: \"ca02c39d-85bd-4cea-ab02-19f67555f81a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrxnl"
Jul 28 10:07:43.356548 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.356529    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/17fe592b-e2cc-42ad-8803-0944d20e4dc0-etc-tuned\") pod \"tuned-vvkrx\" (UID: \"17fe592b-e2cc-42ad-8803-0944d20e4dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-vvkrx"
Jul 28 10:07:43.356548 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.356549    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/675b6659-b757-4eca-ba86-28b21af90501-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-545mm\" (UID: \"675b6659-b757-4eca-ba86-28b21af90501\") " pod="openshift-multus/multus-additional-cni-plugins-545mm"
Jul 28 10:07:43.357015 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.356552    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8490536f-2f40-4389-bf3f-d5cab1ab3a00-konnectivity-ca\") pod \"konnectivity-agent-lqnk6\" (UID: \"8490536f-2f40-4389-bf3f-d5cab1ab3a00\") " pod="kube-system/konnectivity-agent-lqnk6"
Jul 28 10:07:43.357015 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.356565    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/07ad2626-9f01-48ec-adb7-c5f4ab340b7a-original-pull-secret\") pod \"global-pull-secret-syncer-64pm8\" (UID: \"07ad2626-9f01-48ec-adb7-c5f4ab340b7a\") " pod="kube-system/global-pull-secret-syncer-64pm8"
Jul 28 10:07:43.357015 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.356552    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/675b6659-b757-4eca-ba86-28b21af90501-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-545mm\" (UID: \"675b6659-b757-4eca-ba86-28b21af90501\") " pod="openshift-multus/multus-additional-cni-plugins-545mm"
Jul 28 10:07:43.357015 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.356637    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m48lj\" (UniqueName: \"kubernetes.io/projected/ab7f977a-7d91-4d97-85ca-f07a9d373ee8-kube-api-access-m48lj\") pod \"multus-jlf2c\" (UID: \"ab7f977a-7d91-4d97-85ca-f07a9d373ee8\") " pod="openshift-multus/multus-jlf2c"
Jul 28 10:07:43.357015 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.356666    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1a872d0f-af6f-4fd4-bf8f-f763ce610da4-host-slash\") pod \"iptables-alerter-5z6x8\" (UID: \"1a872d0f-af6f-4fd4-bf8f-f763ce610da4\") " pod="openshift-network-operator/iptables-alerter-5z6x8"
Jul 28 10:07:43.357015 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.356690    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ca02c39d-85bd-4cea-ab02-19f67555f81a-host-run-netns\") pod \"ovnkube-node-rrxnl\" (UID: \"ca02c39d-85bd-4cea-ab02-19f67555f81a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrxnl"
Jul 28 10:07:43.357015 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.356713    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab7f977a-7d91-4d97-85ca-f07a9d373ee8-host-var-lib-kubelet\") pod \"multus-jlf2c\" (UID: \"ab7f977a-7d91-4d97-85ca-f07a9d373ee8\") " pod="openshift-multus/multus-jlf2c"
Jul 28 10:07:43.357015 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.356761    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1a872d0f-af6f-4fd4-bf8f-f763ce610da4-host-slash\") pod \"iptables-alerter-5z6x8\" (UID: \"1a872d0f-af6f-4fd4-bf8f-f763ce610da4\") " pod="openshift-network-operator/iptables-alerter-5z6x8"
Jul 28 10:07:43.357015 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.356764    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ca02c39d-85bd-4cea-ab02-19f67555f81a-host-run-netns\") pod \"ovnkube-node-rrxnl\" (UID: \"ca02c39d-85bd-4cea-ab02-19f67555f81a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrxnl"
Jul 28 10:07:43.357015 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.356783    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19db4187-c2f2-46d9-bb21-ce1d3c12c422-metrics-certs\") pod \"network-metrics-daemon-ldq5b\" (UID: \"19db4187-c2f2-46d9-bb21-ce1d3c12c422\") " pod="openshift-multus/network-metrics-daemon-ldq5b"
Jul 28 10:07:43.357015 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.356809    2583 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 28 10:07:43.357015 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.356812    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8490536f-2f40-4389-bf3f-d5cab1ab3a00-agent-certs\") pod \"konnectivity-agent-lqnk6\" (UID: \"8490536f-2f40-4389-bf3f-d5cab1ab3a00\") " pod="kube-system/konnectivity-agent-lqnk6"
Jul 28 10:07:43.357015 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:43.356860    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:07:43.357015 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.356882    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/675b6659-b757-4eca-ba86-28b21af90501-cni-binary-copy\") pod \"multus-additional-cni-plugins-545mm\" (UID: \"675b6659-b757-4eca-ba86-28b21af90501\") " pod="openshift-multus/multus-additional-cni-plugins-545mm"
Jul 28 10:07:43.357015 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:43.356915    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19db4187-c2f2-46d9-bb21-ce1d3c12c422-metrics-certs podName:19db4187-c2f2-46d9-bb21-ce1d3c12c422 nodeName:}" failed. No retries permitted until 2026-07-28 10:07:43.856896287 +0000 UTC m=+2.957402831 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/19db4187-c2f2-46d9-bb21-ce1d3c12c422-metrics-certs") pod "network-metrics-daemon-ldq5b" (UID: "19db4187-c2f2-46d9-bb21-ce1d3c12c422") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:07:43.357015 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.356933    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab7f977a-7d91-4d97-85ca-f07a9d373ee8-etc-kubernetes\") pod \"multus-jlf2c\" (UID: \"ab7f977a-7d91-4d97-85ca-f07a9d373ee8\") " pod="openshift-multus/multus-jlf2c"
Jul 28 10:07:43.357015 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.356958    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/55104d0d-d660-4142-9b73-3277e628eefa-socket-dir\") pod \"aws-ebs-csi-driver-node-vrkpg\" (UID: \"55104d0d-d660-4142-9b73-3277e628eefa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vrkpg"
Jul 28 10:07:43.357776 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.356959    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/675b6659-b757-4eca-ba86-28b21af90501-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-545mm\" (UID: \"675b6659-b757-4eca-ba86-28b21af90501\") " pod="openshift-multus/multus-additional-cni-plugins-545mm"
Jul 28 10:07:43.357776 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.356982    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/17fe592b-e2cc-42ad-8803-0944d20e4dc0-etc-modprobe-d\") pod \"tuned-vvkrx\" (UID: \"17fe592b-e2cc-42ad-8803-0944d20e4dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-vvkrx"
Jul 28 10:07:43.357776 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.357008    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsskl\" (UniqueName: \"kubernetes.io/projected/19db4187-c2f2-46d9-bb21-ce1d3c12c422-kube-api-access-vsskl\") pod \"network-metrics-daemon-ldq5b\" (UID: \"19db4187-c2f2-46d9-bb21-ce1d3c12c422\") " pod="openshift-multus/network-metrics-daemon-ldq5b"
Jul 28 10:07:43.357776 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.357124    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/17fe592b-e2cc-42ad-8803-0944d20e4dc0-etc-modprobe-d\") pod \"tuned-vvkrx\" (UID: \"17fe592b-e2cc-42ad-8803-0944d20e4dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-vvkrx"
Jul 28 10:07:43.357776 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.357151    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/675b6659-b757-4eca-ba86-28b21af90501-tuning-conf-dir\") pod \"multus-additional-cni-plugins-545mm\" (UID: \"675b6659-b757-4eca-ba86-28b21af90501\") " pod="openshift-multus/multus-additional-cni-plugins-545mm"
Jul 28 10:07:43.357776 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.357211    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4jr6\" (UniqueName: \"kubernetes.io/projected/ca02c39d-85bd-4cea-ab02-19f67555f81a-kube-api-access-b4jr6\") pod \"ovnkube-node-rrxnl\" (UID: \"ca02c39d-85bd-4cea-ab02-19f67555f81a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrxnl"
Jul 28 10:07:43.357776 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.357244    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/07ad2626-9f01-48ec-adb7-c5f4ab340b7a-kubelet-config\") pod \"global-pull-secret-syncer-64pm8\" (UID: \"07ad2626-9f01-48ec-adb7-c5f4ab340b7a\") " pod="kube-system/global-pull-secret-syncer-64pm8"
Jul 28 10:07:43.357776 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.357263    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ab7f977a-7d91-4d97-85ca-f07a9d373ee8-host-run-netns\") pod \"multus-jlf2c\" (UID: \"ab7f977a-7d91-4d97-85ca-f07a9d373ee8\") " pod="openshift-multus/multus-jlf2c"
Jul 28 10:07:43.357776 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.357299    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ab7f977a-7d91-4d97-85ca-f07a9d373ee8-host-var-lib-cni-multus\") pod \"multus-jlf2c\" (UID: \"ab7f977a-7d91-4d97-85ca-f07a9d373ee8\") " pod="openshift-multus/multus-jlf2c"
Jul 28 10:07:43.357776 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.357303    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/675b6659-b757-4eca-ba86-28b21af90501-tuning-conf-dir\") pod \"multus-additional-cni-plugins-545mm\" (UID: \"675b6659-b757-4eca-ba86-28b21af90501\") " pod="openshift-multus/multus-additional-cni-plugins-545mm"
Jul 28 10:07:43.357776 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.357335    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ab7f977a-7d91-4d97-85ca-f07a9d373ee8-multus-daemon-config\") pod \"multus-jlf2c\" (UID: \"ab7f977a-7d91-4d97-85ca-f07a9d373ee8\") " pod="openshift-multus/multus-jlf2c"
Jul 28 10:07:43.357776 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.357359    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/675b6659-b757-4eca-ba86-28b21af90501-cni-binary-copy\") pod \"multus-additional-cni-plugins-545mm\" (UID: \"675b6659-b757-4eca-ba86-28b21af90501\") " pod="openshift-multus/multus-additional-cni-plugins-545mm"
Jul 28 10:07:43.357776 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.357361    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ca02c39d-85bd-4cea-ab02-19f67555f81a-ovn-node-metrics-cert\") pod \"ovnkube-node-rrxnl\" (UID: \"ca02c39d-85bd-4cea-ab02-19f67555f81a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrxnl"
Jul 28 10:07:43.357776 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.357429    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl6x8\" (UniqueName: \"kubernetes.io/projected/1a872d0f-af6f-4fd4-bf8f-f763ce610da4-kube-api-access-jl6x8\") pod \"iptables-alerter-5z6x8\" (UID: \"1a872d0f-af6f-4fd4-bf8f-f763ce610da4\") " pod="openshift-network-operator/iptables-alerter-5z6x8"
Jul 28 10:07:43.357776 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.357458    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/17fe592b-e2cc-42ad-8803-0944d20e4dc0-etc-sysctl-d\") pod \"tuned-vvkrx\" (UID: \"17fe592b-e2cc-42ad-8803-0944d20e4dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-vvkrx"
Jul 28 10:07:43.357776 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.357580    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/17fe592b-e2cc-42ad-8803-0944d20e4dc0-etc-sysctl-d\") pod \"tuned-vvkrx\" (UID: \"17fe592b-e2cc-42ad-8803-0944d20e4dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-vvkrx"
Jul 28 10:07:43.357776 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.357581    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca02c39d-85bd-4cea-ab02-19f67555f81a-run-openvswitch\") pod \"ovnkube-node-rrxnl\" (UID: \"ca02c39d-85bd-4cea-ab02-19f67555f81a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrxnl"
Jul 28 10:07:43.358525 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.357609    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ca02c39d-85bd-4cea-ab02-19f67555f81a-ovnkube-script-lib\") pod \"ovnkube-node-rrxnl\" (UID: \"ca02c39d-85bd-4cea-ab02-19f67555f81a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrxnl"
Jul 28 10:07:43.358525 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.357631    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h7nk\" (UniqueName: \"kubernetes.io/projected/42229f5e-156f-4abd-a2ee-700c9f340775-kube-api-access-2h7nk\") pod \"node-resolver-nxpwb\" (UID: \"42229f5e-156f-4abd-a2ee-700c9f340775\") " pod="openshift-dns/node-resolver-nxpwb"
Jul 28 10:07:43.358525 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.357657    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ab7f977a-7d91-4d97-85ca-f07a9d373ee8-multus-cni-dir\") pod \"multus-jlf2c\" (UID: \"ab7f977a-7d91-4d97-85ca-f07a9d373ee8\") " pod="openshift-multus/multus-jlf2c"
Jul 28 10:07:43.358525 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.357676    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ab7f977a-7d91-4d97-85ca-f07a9d373ee8-hostroot\") pod \"multus-jlf2c\" (UID: \"ab7f977a-7d91-4d97-85ca-f07a9d373ee8\") " pod="openshift-multus/multus-jlf2c"
Jul 28 10:07:43.358525 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.357694    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/17fe592b-e2cc-42ad-8803-0944d20e4dc0-sys\") pod \"tuned-vvkrx\" (UID: \"17fe592b-e2cc-42ad-8803-0944d20e4dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-vvkrx"
Jul 28 10:07:43.358525 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.357712    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/17fe592b-e2cc-42ad-8803-0944d20e4dc0-var-lib-kubelet\") pod \"tuned-vvkrx\" (UID: \"17fe592b-e2cc-42ad-8803-0944d20e4dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-vvkrx"
Jul 28 10:07:43.358525 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.357610    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca02c39d-85bd-4cea-ab02-19f67555f81a-run-openvswitch\") pod \"ovnkube-node-rrxnl\" (UID: \"ca02c39d-85bd-4cea-ab02-19f67555f81a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrxnl"
Jul 28 10:07:43.358525 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.357730    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ca02c39d-85bd-4cea-ab02-19f67555f81a-host-slash\") pod \"ovnkube-node-rrxnl\" (UID: \"ca02c39d-85bd-4cea-ab02-19f67555f81a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrxnl"
Jul 28 10:07:43.358525 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.357764    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ca02c39d-85bd-4cea-ab02-19f67555f81a-log-socket\") pod \"ovnkube-node-rrxnl\" (UID: \"ca02c39d-85bd-4cea-ab02-19f67555f81a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrxnl"
Jul 28 10:07:43.358525 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.357770    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ca02c39d-85bd-4cea-ab02-19f67555f81a-host-slash\") pod \"ovnkube-node-rrxnl\" (UID: \"ca02c39d-85bd-4cea-ab02-19f67555f81a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrxnl"
Jul 28 10:07:43.358525 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.357791    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ca02c39d-85bd-4cea-ab02-19f67555f81a-env-overrides\") pod \"ovnkube-node-rrxnl\" (UID: \"ca02c39d-85bd-4cea-ab02-19f67555f81a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrxnl"
Jul 28 10:07:43.358525 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.357812    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/17fe592b-e2cc-42ad-8803-0944d20e4dc0-sys\") pod \"tuned-vvkrx\" (UID: \"17fe592b-e2cc-42ad-8803-0944d20e4dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-vvkrx"
Jul 28 10:07:43.358525 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.357816    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/55104d0d-d660-4142-9b73-3277e628eefa-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vrkpg\" (UID: \"55104d0d-d660-4142-9b73-3277e628eefa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vrkpg"
Jul 28 10:07:43.358525 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.357832    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/eeccf40b-0096-40d3-b231-d7ce00688cc9-serviceca\") pod \"node-ca-v94ww\" (UID: \"eeccf40b-0096-40d3-b231-d7ce00688cc9\") " pod="openshift-image-registry/node-ca-v94ww"
Jul 28 10:07:43.358525 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.357848    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fzw8\" (UniqueName: \"kubernetes.io/projected/eeccf40b-0096-40d3-b231-d7ce00688cc9-kube-api-access-2fzw8\") pod \"node-ca-v94ww\" (UID: \"eeccf40b-0096-40d3-b231-d7ce00688cc9\") " pod="openshift-image-registry/node-ca-v94ww"
Jul 28 10:07:43.358525 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.357851    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/17fe592b-e2cc-42ad-8803-0944d20e4dc0-var-lib-kubelet\") pod \"tuned-vvkrx\" (UID: \"17fe592b-e2cc-42ad-8803-0944d20e4dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-vvkrx"
Jul 28 10:07:43.358525 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.357862    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/675b6659-b757-4eca-ba86-28b21af90501-system-cni-dir\") pod \"multus-additional-cni-plugins-545mm\" (UID: \"675b6659-b757-4eca-ba86-28b21af90501\") " pod="openshift-multus/multus-additional-cni-plugins-545mm"
Jul 28 10:07:43.359361 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.357878    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ab7f977a-7d91-4d97-85ca-f07a9d373ee8-cnibin\") pod \"multus-jlf2c\" (UID: \"ab7f977a-7d91-4d97-85ca-f07a9d373ee8\") " pod="openshift-multus/multus-jlf2c"
Jul 28 10:07:43.359361 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.357892    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/55104d0d-d660-4142-9b73-3277e628eefa-device-dir\") pod \"aws-ebs-csi-driver-node-vrkpg\" (UID: \"55104d0d-d660-4142-9b73-3277e628eefa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vrkpg"
Jul 28 10:07:43.359361 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.357906    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/55104d0d-d660-4142-9b73-3277e628eefa-sys-fs\") pod \"aws-ebs-csi-driver-node-vrkpg\" (UID: \"55104d0d-d660-4142-9b73-3277e628eefa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vrkpg"
Jul 28 10:07:43.359361 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.357936    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/17fe592b-e2cc-42ad-8803-0944d20e4dc0-etc-sysconfig\") pod \"tuned-vvkrx\" (UID: \"17fe592b-e2cc-42ad-8803-0944d20e4dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-vvkrx"
Jul 28 10:07:43.359361 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.357952    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/17fe592b-e2cc-42ad-8803-0944d20e4dc0-etc-kubernetes\") pod \"tuned-vvkrx\" (UID: \"17fe592b-e2cc-42ad-8803-0944d20e4dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-vvkrx"
Jul 28 10:07:43.359361 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.357965    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/07ad2626-9f01-48ec-adb7-c5f4ab340b7a-dbus\") pod \"global-pull-secret-syncer-64pm8\" (UID: \"07ad2626-9f01-48ec-adb7-c5f4ab340b7a\") " pod="kube-system/global-pull-secret-syncer-64pm8"
Jul 28 10:07:43.359361 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.357980    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ab7f977a-7d91-4d97-85ca-f07a9d373ee8-os-release\") pod \"multus-jlf2c\" (UID: \"ab7f977a-7d91-4d97-85ca-f07a9d373ee8\") " pod="openshift-multus/multus-jlf2c"
Jul 28 10:07:43.359361 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.357993    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/55104d0d-d660-4142-9b73-3277e628eefa-etc-selinux\") pod \"aws-ebs-csi-driver-node-vrkpg\" (UID: \"55104d0d-d660-4142-9b73-3277e628eefa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vrkpg"
Jul 28 10:07:43.359361 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.358016    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1a872d0f-af6f-4fd4-bf8f-f763ce610da4-iptables-alerter-script\") pod \"iptables-alerter-5z6x8\" (UID: \"1a872d0f-af6f-4fd4-bf8f-f763ce610da4\") " pod="openshift-network-operator/iptables-alerter-5z6x8"
Jul 28 10:07:43.359361 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.358033    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca02c39d-85bd-4cea-ab02-19f67555f81a-host-run-ovn-kubernetes\") pod \"ovnkube-node-rrxnl\" (UID: \"ca02c39d-85bd-4cea-ab02-19f67555f81a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrxnl"
Jul 28 10:07:43.359361 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.358048    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ca02c39d-85bd-4cea-ab02-19f67555f81a-host-cni-bin\") pod \"ovnkube-node-rrxnl\" (UID: \"ca02c39d-85bd-4cea-ab02-19f67555f81a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrxnl"
Jul 28 10:07:43.359361 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.358082    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ab7f977a-7d91-4d97-85ca-f07a9d373ee8-multus-socket-dir-parent\") pod \"multus-jlf2c\" (UID: \"ab7f977a-7d91-4d97-85ca-f07a9d373ee8\") " pod="openshift-multus/multus-jlf2c"
Jul 28 10:07:43.359361 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.358105    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ab7f977a-7d91-4d97-85ca-f07a9d373ee8-host-run-multus-certs\") pod \"multus-jlf2c\" (UID: \"ab7f977a-7d91-4d97-85ca-f07a9d373ee8\") " pod="openshift-multus/multus-jlf2c"
Jul 28 10:07:43.359361 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.358126    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/675b6659-b757-4eca-ba86-28b21af90501-cnibin\") pod \"multus-additional-cni-plugins-545mm\" (UID: \"675b6659-b757-4eca-ba86-28b21af90501\") " pod="openshift-multus/multus-additional-cni-plugins-545mm"
Jul 28 10:07:43.359361 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.358141    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca02c39d-85bd-4cea-ab02-19f67555f81a-etc-openvswitch\") pod \"ovnkube-node-rrxnl\" (UID: \"ca02c39d-85bd-4cea-ab02-19f67555f81a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrxnl"
Jul 28 10:07:43.359361 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.358156    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ab7f977a-7d91-4d97-85ca-f07a9d373ee8-multus-conf-dir\") pod \"multus-jlf2c\" (UID: \"ab7f977a-7d91-4d97-85ca-f07a9d373ee8\") " pod="openshift-multus/multus-jlf2c"
Jul 28 10:07:43.359361 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.358171    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/17fe592b-e2cc-42ad-8803-0944d20e4dc0-run\") pod \"tuned-vvkrx\" (UID: \"17fe592b-e2cc-42ad-8803-0944d20e4dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-vvkrx"
Jul 28 10:07:43.360138 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.358209    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/675b6659-b757-4eca-ba86-28b21af90501-os-release\") pod \"multus-additional-cni-plugins-545mm\" (UID: \"675b6659-b757-4eca-ba86-28b21af90501\") " pod="openshift-multus/multus-additional-cni-plugins-545mm"
Jul 28 10:07:43.360138 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.358233    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca02c39d-85bd-4cea-ab02-19f67555f81a-host-kubelet\") pod \"ovnkube-node-rrxnl\" (UID: \"ca02c39d-85bd-4cea-ab02-19f67555f81a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrxnl"
Jul 28 10:07:43.360138 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.358256    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ca02c39d-85bd-4cea-ab02-19f67555f81a-systemd-units\") pod \"ovnkube-node-rrxnl\" (UID: \"ca02c39d-85bd-4cea-ab02-19f67555f81a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrxnl"
Jul 28 10:07:43.360138 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.358280    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ab7f977a-7d91-4d97-85ca-f07a9d373ee8-system-cni-dir\") pod \"multus-jlf2c\" (UID: \"ab7f977a-7d91-4d97-85ca-f07a9d373ee8\") " pod="openshift-multus/multus-jlf2c"
Jul 28 10:07:43.360138 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.358302    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ab7f977a-7d91-4d97-85ca-f07a9d373ee8-cni-binary-copy\") pod \"multus-jlf2c\" (UID: \"ab7f977a-7d91-4d97-85ca-f07a9d373ee8\") " pod="openshift-multus/multus-jlf2c"
Jul 28 10:07:43.360138 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.358327    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/17fe592b-e2cc-42ad-8803-0944d20e4dc0-etc-sysctl-conf\") pod \"tuned-vvkrx\" (UID: \"17fe592b-e2cc-42ad-8803-0944d20e4dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-vvkrx"
Jul 28 10:07:43.360138 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.358358    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ca02c39d-85bd-4cea-ab02-19f67555f81a-host-cni-netd\") pod \"ovnkube-node-rrxnl\" (UID: \"ca02c39d-85bd-4cea-ab02-19f67555f81a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrxnl"
Jul 28 10:07:43.360138 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.358384    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/17fe592b-e2cc-42ad-8803-0944d20e4dc0-lib-modules\") pod \"tuned-vvkrx\" (UID: \"17fe592b-e2cc-42ad-8803-0944d20e4dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-vvkrx"
Jul 28 10:07:43.360138 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.358407    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17fe592b-e2cc-42ad-8803-0944d20e4dc0-tmp\") pod \"tuned-vvkrx\" (UID: \"17fe592b-e2cc-42ad-8803-0944d20e4dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-vvkrx"
Jul 28 10:07:43.360138 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.358435    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw9j2\" (UniqueName: \"kubernetes.io/projected/675b6659-b757-4eca-ba86-28b21af90501-kube-api-access-hw9j2\") pod \"multus-additional-cni-plugins-545mm\" (UID: \"675b6659-b757-4eca-ba86-28b21af90501\") " pod="openshift-multus/multus-additional-cni-plugins-545mm"
Jul 28 10:07:43.360138 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.358463    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ca02c39d-85bd-4cea-ab02-19f67555f81a-run-systemd\") pod \"ovnkube-node-rrxnl\" (UID: \"ca02c39d-85bd-4cea-ab02-19f67555f81a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrxnl"
Jul 28 10:07:43.360138 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.358479    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/eeccf40b-0096-40d3-b231-d7ce00688cc9-serviceca\") pod \"node-ca-v94ww\" (UID: \"eeccf40b-0096-40d3-b231-d7ce00688cc9\") " pod="openshift-image-registry/node-ca-v94ww"
Jul 28 10:07:43.360138 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.358489    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ca02c39d-85bd-4cea-ab02-19f67555f81a-ovnkube-config\") pod \"ovnkube-node-rrxnl\" (UID: \"ca02c39d-85bd-4cea-ab02-19f67555f81a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrxnl"
Jul 28 10:07:43.360138 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.358539    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ab7f977a-7d91-4d97-85ca-f07a9d373ee8-host-run-k8s-cni-cncf-io\") pod \"multus-jlf2c\" (UID: \"ab7f977a-7d91-4d97-85ca-f07a9d373ee8\") " pod="openshift-multus/multus-jlf2c"
Jul 28 10:07:43.360138 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.358639    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/675b6659-b757-4eca-ba86-28b21af90501-system-cni-dir\") pod \"multus-additional-cni-plugins-545mm\" (UID: \"675b6659-b757-4eca-ba86-28b21af90501\") " pod="openshift-multus/multus-additional-cni-plugins-545mm"
Jul 28 10:07:43.360138 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.357893    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ca02c39d-85bd-4cea-ab02-19f67555f81a-log-socket\") pod \"ovnkube-node-rrxnl\" (UID: \"ca02c39d-85bd-4cea-ab02-19f67555f81a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrxnl"
Jul 28 10:07:43.360138 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.358696    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ca02c39d-85bd-4cea-ab02-19f67555f81a-ovnkube-script-lib\") pod \"ovnkube-node-rrxnl\" (UID: \"ca02c39d-85bd-4cea-ab02-19f67555f81a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrxnl"
Jul 28 10:07:43.360961 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.358727    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/17fe592b-e2cc-42ad-8803-0944d20e4dc0-etc-sysconfig\") pod \"tuned-vvkrx\" (UID: \"17fe592b-e2cc-42ad-8803-0944d20e4dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-vvkrx"
Jul 28 10:07:43.360961 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.358776    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/17fe592b-e2cc-42ad-8803-0944d20e4dc0-etc-kubernetes\") pod \"tuned-vvkrx\" (UID: \"17fe592b-e2cc-42ad-8803-0944d20e4dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-vvkrx"
Jul 28 10:07:43.360961 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.358848    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ca02c39d-85bd-4cea-ab02-19f67555f81a-host-cni-bin\") pod \"ovnkube-node-rrxnl\" (UID: \"ca02c39d-85bd-4cea-ab02-19f67555f81a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrxnl"
Jul 28 10:07:43.360961 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.358893    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca02c39d-85bd-4cea-ab02-19f67555f81a-host-run-ovn-kubernetes\") pod \"ovnkube-node-rrxnl\" (UID: \"ca02c39d-85bd-4cea-ab02-19f67555f81a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrxnl"
Jul 28 10:07:43.360961 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.358906    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ca02c39d-85bd-4cea-ab02-19f67555f81a-ovnkube-config\") pod \"ovnkube-node-rrxnl\" (UID: \"ca02c39d-85bd-4cea-ab02-19f67555f81a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrxnl"
Jul 28 10:07:43.360961 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.358952    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/675b6659-b757-4eca-ba86-28b21af90501-cnibin\") pod \"multus-additional-cni-plugins-545mm\" (UID: \"675b6659-b757-4eca-ba86-28b21af90501\") " pod="openshift-multus/multus-additional-cni-plugins-545mm"
Jul 28 10:07:43.360961 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.358986    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca02c39d-85bd-4cea-ab02-19f67555f81a-etc-openvswitch\") pod \"ovnkube-node-rrxnl\" (UID: \"ca02c39d-85bd-4cea-ab02-19f67555f81a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrxnl"
Jul 28 10:07:43.360961 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.359039    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/17fe592b-e2cc-42ad-8803-0944d20e4dc0-run\") pod \"tuned-vvkrx\" (UID: \"17fe592b-e2cc-42ad-8803-0944d20e4dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-vvkrx"
Jul 28 10:07:43.360961 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.359043    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/17fe592b-e2cc-42ad-8803-0944d20e4dc0-etc-sysctl-conf\") pod \"tuned-vvkrx\" (UID: \"17fe592b-e2cc-42ad-8803-0944d20e4dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-vvkrx"
Jul 28 10:07:43.360961 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.359094    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/675b6659-b757-4eca-ba86-28b21af90501-os-release\") pod \"multus-additional-cni-plugins-545mm\" (UID: \"675b6659-b757-4eca-ba86-28b21af90501\") " pod="openshift-multus/multus-additional-cni-plugins-545mm"
Jul 28 10:07:43.360961 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.359103    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ca02c39d-85bd-4cea-ab02-19f67555f81a-host-cni-netd\") pod \"ovnkube-node-rrxnl\" (UID: \"ca02c39d-85bd-4cea-ab02-19f67555f81a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrxnl"
Jul 28 10:07:43.360961 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.359166    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ca02c39d-85bd-4cea-ab02-19f67555f81a-systemd-units\") pod \"ovnkube-node-rrxnl\" (UID: \"ca02c39d-85bd-4cea-ab02-19f67555f81a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrxnl"
Jul 28 10:07:43.360961 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.359233    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1a872d0f-af6f-4fd4-bf8f-f763ce610da4-iptables-alerter-script\") pod \"iptables-alerter-5z6x8\" (UID: \"1a872d0f-af6f-4fd4-bf8f-f763ce610da4\") " pod="openshift-network-operator/iptables-alerter-5z6x8"
Jul 28 10:07:43.360961 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.359399    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ca02c39d-85bd-4cea-ab02-19f67555f81a-run-systemd\") pod \"ovnkube-node-rrxnl\" (UID: \"ca02c39d-85bd-4cea-ab02-19f67555f81a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrxnl"
Jul 28 10:07:43.360961 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.359408    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca02c39d-85bd-4cea-ab02-19f67555f81a-host-kubelet\") pod \"ovnkube-node-rrxnl\" (UID: \"ca02c39d-85bd-4cea-ab02-19f67555f81a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrxnl"
Jul 28 10:07:43.360961 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.359442    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/17fe592b-e2cc-42ad-8803-0944d20e4dc0-lib-modules\") pod \"tuned-vvkrx\" (UID: \"17fe592b-e2cc-42ad-8803-0944d20e4dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-vvkrx"
Jul 28 10:07:43.360961 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.359640    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ca02c39d-85bd-4cea-ab02-19f67555f81a-env-overrides\") pod \"ovnkube-node-rrxnl\" (UID: \"ca02c39d-85bd-4cea-ab02-19f67555f81a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrxnl"
Jul 28 10:07:43.360961 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.360741    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/17fe592b-e2cc-42ad-8803-0944d20e4dc0-etc-tuned\") pod \"tuned-vvkrx\" (UID: \"17fe592b-e2cc-42ad-8803-0944d20e4dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-vvkrx"
Jul 28 10:07:43.363723 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.363683    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17fe592b-e2cc-42ad-8803-0944d20e4dc0-tmp\") pod \"tuned-vvkrx\" (UID: \"17fe592b-e2cc-42ad-8803-0944d20e4dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-vvkrx"
Jul 28 10:07:43.365499 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:43.365316    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 10:07:43.365499 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:43.365342    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 10:07:43.365499 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:43.365355    2583 projected.go:196] Error preparing data for projected volume kube-api-access-2gzlj for pod openshift-network-diagnostics/network-check-target-7dfzm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:07:43.365499 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:43.365424    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d329552c-d2a3-4960-9130-bc9e211efc18-kube-api-access-2gzlj podName:d329552c-d2a3-4960-9130-bc9e211efc18 nodeName:}" failed. No retries permitted until 2026-07-28 10:07:43.865397092 +0000 UTC m=+2.965903627 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-2gzlj" (UniqueName: "kubernetes.io/projected/d329552c-d2a3-4960-9130-bc9e211efc18-kube-api-access-2gzlj") pod "network-check-target-7dfzm" (UID: "d329552c-d2a3-4960-9130-bc9e211efc18") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:07:43.365499 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.365414    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8490536f-2f40-4389-bf3f-d5cab1ab3a00-agent-certs\") pod \"konnectivity-agent-lqnk6\" (UID: \"8490536f-2f40-4389-bf3f-d5cab1ab3a00\") " pod="kube-system/konnectivity-agent-lqnk6"
Jul 28 10:07:43.365834 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.365593    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ca02c39d-85bd-4cea-ab02-19f67555f81a-ovn-node-metrics-cert\") pod \"ovnkube-node-rrxnl\" (UID: \"ca02c39d-85bd-4cea-ab02-19f67555f81a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrxnl"
Jul 28 10:07:43.372603 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.372583    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbvk7\" (UniqueName: \"kubernetes.io/projected/17fe592b-e2cc-42ad-8803-0944d20e4dc0-kube-api-access-jbvk7\") pod \"tuned-vvkrx\" (UID: \"17fe592b-e2cc-42ad-8803-0944d20e4dc0\") " pod="openshift-cluster-node-tuning-operator/tuned-vvkrx"
Jul 28 10:07:43.372695 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.372627    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsskl\" (UniqueName: \"kubernetes.io/projected/19db4187-c2f2-46d9-bb21-ce1d3c12c422-kube-api-access-vsskl\") pod \"network-metrics-daemon-ldq5b\" (UID: \"19db4187-c2f2-46d9-bb21-ce1d3c12c422\") " pod="openshift-multus/network-metrics-daemon-ldq5b"
Jul 28 10:07:43.373534 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.373098    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4jr6\" (UniqueName: \"kubernetes.io/projected/ca02c39d-85bd-4cea-ab02-19f67555f81a-kube-api-access-b4jr6\") pod \"ovnkube-node-rrxnl\" (UID: \"ca02c39d-85bd-4cea-ab02-19f67555f81a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrxnl"
Jul 28 10:07:43.373624 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.373538    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fzw8\" (UniqueName: \"kubernetes.io/projected/eeccf40b-0096-40d3-b231-d7ce00688cc9-kube-api-access-2fzw8\") pod \"node-ca-v94ww\" (UID: \"eeccf40b-0096-40d3-b231-d7ce00688cc9\") " pod="openshift-image-registry/node-ca-v94ww"
Jul 28 10:07:43.373624 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.373575    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl6x8\" (UniqueName: \"kubernetes.io/projected/1a872d0f-af6f-4fd4-bf8f-f763ce610da4-kube-api-access-jl6x8\") pod \"iptables-alerter-5z6x8\" (UID: \"1a872d0f-af6f-4fd4-bf8f-f763ce610da4\") " pod="openshift-network-operator/iptables-alerter-5z6x8"
Jul 28 10:07:43.375899 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.375846    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw9j2\" (UniqueName: \"kubernetes.io/projected/675b6659-b757-4eca-ba86-28b21af90501-kube-api-access-hw9j2\") pod \"multus-additional-cni-plugins-545mm\" (UID: \"675b6659-b757-4eca-ba86-28b21af90501\") " pod="openshift-multus/multus-additional-cni-plugins-545mm"
Jul 28 10:07:43.437329 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.437276    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-110.ec2.internal" event={"ID":"c2a2b445e0df91dcaba3ba620484e779","Type":"ContainerStarted","Data":"6d5eb2b15c798724048d50ec43baff6ec4c514380ea8eada9a4b1962777c1319"}
Jul 28 10:07:43.438764 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.438734    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-110.ec2.internal" event={"ID":"2b84527c672e7fff4b21007a2e0dc2c6","Type":"ContainerStarted","Data":"6266dafceb756cec826760f6c07786cab08c997a20c7296a6795029d778a6c06"}
Jul 28 10:07:43.459020 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.458979    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/07ad2626-9f01-48ec-adb7-c5f4ab340b7a-kubelet-config\") pod \"global-pull-secret-syncer-64pm8\" (UID: \"07ad2626-9f01-48ec-adb7-c5f4ab340b7a\") " pod="kube-system/global-pull-secret-syncer-64pm8"
Jul 28 10:07:43.459020 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.459021    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ab7f977a-7d91-4d97-85ca-f07a9d373ee8-host-run-netns\") pod \"multus-jlf2c\" (UID: \"ab7f977a-7d91-4d97-85ca-f07a9d373ee8\") " pod="openshift-multus/multus-jlf2c"
Jul 28 10:07:43.459204 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.459047    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ab7f977a-7d91-4d97-85ca-f07a9d373ee8-host-var-lib-cni-multus\") pod \"multus-jlf2c\" (UID: \"ab7f977a-7d91-4d97-85ca-f07a9d373ee8\") " pod="openshift-multus/multus-jlf2c"
Jul 28 10:07:43.459204 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.459046    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/07ad2626-9f01-48ec-adb7-c5f4ab340b7a-kubelet-config\") pod \"global-pull-secret-syncer-64pm8\" (UID: \"07ad2626-9f01-48ec-adb7-c5f4ab340b7a\") " pod="kube-system/global-pull-secret-syncer-64pm8"
Jul 28 10:07:43.459204 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.459117    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ab7f977a-7d91-4d97-85ca-f07a9d373ee8-host-var-lib-cni-multus\") pod \"multus-jlf2c\" (UID: \"ab7f977a-7d91-4d97-85ca-f07a9d373ee8\") " pod="openshift-multus/multus-jlf2c"
Jul 28 10:07:43.459204 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.459079    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ab7f977a-7d91-4d97-85ca-f07a9d373ee8-host-run-netns\") pod \"multus-jlf2c\" (UID: \"ab7f977a-7d91-4d97-85ca-f07a9d373ee8\") " pod="openshift-multus/multus-jlf2c"
Jul 28 10:07:43.459204 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.459071    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ab7f977a-7d91-4d97-85ca-f07a9d373ee8-multus-daemon-config\") pod \"multus-jlf2c\" (UID: \"ab7f977a-7d91-4d97-85ca-f07a9d373ee8\") " pod="openshift-multus/multus-jlf2c"
Jul 28 10:07:43.459444 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.459219    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h7nk\" (UniqueName: \"kubernetes.io/projected/42229f5e-156f-4abd-a2ee-700c9f340775-kube-api-access-2h7nk\") pod \"node-resolver-nxpwb\" (UID: \"42229f5e-156f-4abd-a2ee-700c9f340775\") " pod="openshift-dns/node-resolver-nxpwb"
Jul 28 10:07:43.459444 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.459246    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ab7f977a-7d91-4d97-85ca-f07a9d373ee8-multus-cni-dir\") pod \"multus-jlf2c\" (UID: \"ab7f977a-7d91-4d97-85ca-f07a9d373ee8\") " pod="openshift-multus/multus-jlf2c"
Jul 28 10:07:43.459444 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.459278    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ab7f977a-7d91-4d97-85ca-f07a9d373ee8-hostroot\") pod \"multus-jlf2c\" (UID: \"ab7f977a-7d91-4d97-85ca-f07a9d373ee8\") " pod="openshift-multus/multus-jlf2c"
Jul 28 10:07:43.459444 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.459307    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/55104d0d-d660-4142-9b73-3277e628eefa-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vrkpg\" (UID: \"55104d0d-d660-4142-9b73-3277e628eefa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vrkpg"
Jul 28 10:07:43.459444 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.459344    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ab7f977a-7d91-4d97-85ca-f07a9d373ee8-cnibin\") pod \"multus-jlf2c\" (UID: \"ab7f977a-7d91-4d97-85ca-f07a9d373ee8\") " pod="openshift-multus/multus-jlf2c"
Jul 28 10:07:43.459444 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.459365    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/55104d0d-d660-4142-9b73-3277e628eefa-device-dir\") pod \"aws-ebs-csi-driver-node-vrkpg\" (UID: \"55104d0d-d660-4142-9b73-3277e628eefa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vrkpg"
Jul 28 10:07:43.459444 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.459385    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/55104d0d-d660-4142-9b73-3277e628eefa-sys-fs\") pod \"aws-ebs-csi-driver-node-vrkpg\" (UID: \"55104d0d-d660-4142-9b73-3277e628eefa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vrkpg"
Jul 28 10:07:43.459444 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.459410    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/07ad2626-9f01-48ec-adb7-c5f4ab340b7a-dbus\") pod \"global-pull-secret-syncer-64pm8\" (UID: \"07ad2626-9f01-48ec-adb7-c5f4ab340b7a\") " pod="kube-system/global-pull-secret-syncer-64pm8"
Jul 28 10:07:43.459444 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.459420    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ab7f977a-7d91-4d97-85ca-f07a9d373ee8-multus-cni-dir\") pod \"multus-jlf2c\" (UID: \"ab7f977a-7d91-4d97-85ca-f07a9d373ee8\") " pod="openshift-multus/multus-jlf2c"
Jul 28 10:07:43.459696 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.459453    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ab7f977a-7d91-4d97-85ca-f07a9d373ee8-os-release\") pod \"multus-jlf2c\" (UID: \"ab7f977a-7d91-4d97-85ca-f07a9d373ee8\") " pod="openshift-multus/multus-jlf2c"
Jul 28 10:07:43.459696 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.459479    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/55104d0d-d660-4142-9b73-3277e628eefa-etc-selinux\") pod \"aws-ebs-csi-driver-node-vrkpg\" (UID: \"55104d0d-d660-4142-9b73-3277e628eefa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vrkpg"
Jul 28 10:07:43.459696 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.459476    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ab7f977a-7d91-4d97-85ca-f07a9d373ee8-cnibin\") pod \"multus-jlf2c\" (UID: \"ab7f977a-7d91-4d97-85ca-f07a9d373ee8\") " pod="openshift-multus/multus-jlf2c"
Jul 28 10:07:43.459696 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.459530    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/55104d0d-d660-4142-9b73-3277e628eefa-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vrkpg\" (UID: \"55104d0d-d660-4142-9b73-3277e628eefa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vrkpg"
Jul 28 10:07:43.459696 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.459540    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/55104d0d-d660-4142-9b73-3277e628eefa-sys-fs\") pod \"aws-ebs-csi-driver-node-vrkpg\" (UID: \"55104d0d-d660-4142-9b73-3277e628eefa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vrkpg"
Jul 28 10:07:43.459696 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.459586    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ab7f977a-7d91-4d97-85ca-f07a9d373ee8-hostroot\") pod \"multus-jlf2c\" (UID: \"ab7f977a-7d91-4d97-85ca-f07a9d373ee8\") " pod="openshift-multus/multus-jlf2c"
Jul 28 10:07:43.459696 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.459593    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ab7f977a-7d91-4d97-85ca-f07a9d373ee8-multus-daemon-config\") pod \"multus-jlf2c\" (UID: \"ab7f977a-7d91-4d97-85ca-f07a9d373ee8\") " pod="openshift-multus/multus-jlf2c"
Jul 28 10:07:43.459696 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.459607    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/55104d0d-d660-4142-9b73-3277e628eefa-etc-selinux\") pod \"aws-ebs-csi-driver-node-vrkpg\" (UID: \"55104d0d-d660-4142-9b73-3277e628eefa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vrkpg"
Jul 28 10:07:43.459696 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.459608    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/07ad2626-9f01-48ec-adb7-c5f4ab340b7a-dbus\") pod \"global-pull-secret-syncer-64pm8\" (UID: \"07ad2626-9f01-48ec-adb7-c5f4ab340b7a\") " pod="kube-system/global-pull-secret-syncer-64pm8"
Jul 28 10:07:43.459696 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.459633    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ab7f977a-7d91-4d97-85ca-f07a9d373ee8-multus-socket-dir-parent\") pod \"multus-jlf2c\" (UID: \"ab7f977a-7d91-4d97-85ca-f07a9d373ee8\") " pod="openshift-multus/multus-jlf2c"
Jul 28 10:07:43.459696 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.459635    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/55104d0d-d660-4142-9b73-3277e628eefa-device-dir\") pod \"aws-ebs-csi-driver-node-vrkpg\" (UID: \"55104d0d-d660-4142-9b73-3277e628eefa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vrkpg"
Jul 28 10:07:43.459696 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.459661    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ab7f977a-7d91-4d97-85ca-f07a9d373ee8-os-release\") pod \"multus-jlf2c\" (UID: \"ab7f977a-7d91-4d97-85ca-f07a9d373ee8\") " pod="openshift-multus/multus-jlf2c"
Jul 28 10:07:43.459696 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.459664    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ab7f977a-7d91-4d97-85ca-f07a9d373ee8-host-run-multus-certs\") pod \"multus-jlf2c\" (UID: \"ab7f977a-7d91-4d97-85ca-f07a9d373ee8\") " pod="openshift-multus/multus-jlf2c"
Jul 28 10:07:43.459696 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.459662    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ab7f977a-7d91-4d97-85ca-f07a9d373ee8-multus-socket-dir-parent\") pod \"multus-jlf2c\" (UID: \"ab7f977a-7d91-4d97-85ca-f07a9d373ee8\") " pod="openshift-multus/multus-jlf2c"
Jul 28 10:07:43.459696 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.459699    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ab7f977a-7d91-4d97-85ca-f07a9d373ee8-multus-conf-dir\") pod \"multus-jlf2c\" (UID: \"ab7f977a-7d91-4d97-85ca-f07a9d373ee8\") " pod="openshift-multus/multus-jlf2c"
Jul 28 10:07:43.460417 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.459719    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ab7f977a-7d91-4d97-85ca-f07a9d373ee8-system-cni-dir\") pod \"multus-jlf2c\" (UID: \"ab7f977a-7d91-4d97-85ca-f07a9d373ee8\") " pod="openshift-multus/multus-jlf2c"
Jul 28 10:07:43.460417 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.459699    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ab7f977a-7d91-4d97-85ca-f07a9d373ee8-host-run-multus-certs\") pod \"multus-jlf2c\" (UID: \"ab7f977a-7d91-4d97-85ca-f07a9d373ee8\") " pod="openshift-multus/multus-jlf2c"
Jul 28 10:07:43.460417 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.459765    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ab7f977a-7d91-4d97-85ca-f07a9d373ee8-cni-binary-copy\") pod \"multus-jlf2c\" (UID: \"ab7f977a-7d91-4d97-85ca-f07a9d373ee8\") " pod="openshift-multus/multus-jlf2c"
Jul 28 10:07:43.460417 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.459760    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ab7f977a-7d91-4d97-85ca-f07a9d373ee8-multus-conf-dir\") pod \"multus-jlf2c\" (UID: \"ab7f977a-7d91-4d97-85ca-f07a9d373ee8\") " pod="openshift-multus/multus-jlf2c"
Jul 28 10:07:43.460417 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.459791    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ab7f977a-7d91-4d97-85ca-f07a9d373ee8-system-cni-dir\") pod \"multus-jlf2c\" (UID: \"ab7f977a-7d91-4d97-85ca-f07a9d373ee8\") " pod="openshift-multus/multus-jlf2c"
Jul 28 10:07:43.460417 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.459811    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ab7f977a-7d91-4d97-85ca-f07a9d373ee8-host-run-k8s-cni-cncf-io\") pod \"multus-jlf2c\" (UID: \"ab7f977a-7d91-4d97-85ca-f07a9d373ee8\") " pod="openshift-multus/multus-jlf2c"
Jul 28 10:07:43.460417 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.459839    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ab7f977a-7d91-4d97-85ca-f07a9d373ee8-host-var-lib-cni-bin\") pod \"multus-jlf2c\" (UID: \"ab7f977a-7d91-4d97-85ca-f07a9d373ee8\") " pod="openshift-multus/multus-jlf2c"
Jul 28 10:07:43.460417 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.459855    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42229f5e-156f-4abd-a2ee-700c9f340775-tmp-dir\") pod \"node-resolver-nxpwb\" (UID: \"42229f5e-156f-4abd-a2ee-700c9f340775\") " pod="openshift-dns/node-resolver-nxpwb"
Jul 28 10:07:43.460417 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.459870    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd8b4\" (UniqueName: \"kubernetes.io/projected/55104d0d-d660-4142-9b73-3277e628eefa-kube-api-access-kd8b4\") pod \"aws-ebs-csi-driver-node-vrkpg\" (UID: \"55104d0d-d660-4142-9b73-3277e628eefa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vrkpg"
Jul 28 10:07:43.460417 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.459894    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ab7f977a-7d91-4d97-85ca-f07a9d373ee8-host-var-lib-cni-bin\") pod \"multus-jlf2c\" (UID: \"ab7f977a-7d91-4d97-85ca-f07a9d373ee8\") " pod="openshift-multus/multus-jlf2c"
Jul 28 10:07:43.460417 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.459910    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/42229f5e-156f-4abd-a2ee-700c9f340775-hosts-file\") pod \"node-resolver-nxpwb\" (UID: \"42229f5e-156f-4abd-a2ee-700c9f340775\") " pod="openshift-dns/node-resolver-nxpwb"
Jul 28 10:07:43.460417 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.459925    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ab7f977a-7d91-4d97-85ca-f07a9d373ee8-host-run-k8s-cni-cncf-io\") pod \"multus-jlf2c\" (UID: \"ab7f977a-7d91-4d97-85ca-f07a9d373ee8\") " pod="openshift-multus/multus-jlf2c"
Jul 28 10:07:43.460417 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.459933    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/55104d0d-d660-4142-9b73-3277e628eefa-registration-dir\") pod \"aws-ebs-csi-driver-node-vrkpg\" (UID: \"55104d0d-d660-4142-9b73-3277e628eefa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vrkpg"
Jul 28 10:07:43.460417 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.459959    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/07ad2626-9f01-48ec-adb7-c5f4ab340b7a-original-pull-secret\") pod \"global-pull-secret-syncer-64pm8\" (UID: \"07ad2626-9f01-48ec-adb7-c5f4ab340b7a\") " pod="kube-system/global-pull-secret-syncer-64pm8"
Jul 28 10:07:43.460417 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.459985    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m48lj\" (UniqueName: \"kubernetes.io/projected/ab7f977a-7d91-4d97-85ca-f07a9d373ee8-kube-api-access-m48lj\") pod \"multus-jlf2c\" (UID: \"ab7f977a-7d91-4d97-85ca-f07a9d373ee8\") " pod="openshift-multus/multus-jlf2c"
Jul 28 10:07:43.460417 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.460013    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab7f977a-7d91-4d97-85ca-f07a9d373ee8-host-var-lib-kubelet\") pod \"multus-jlf2c\" (UID: \"ab7f977a-7d91-4d97-85ca-f07a9d373ee8\") " pod="openshift-multus/multus-jlf2c"
Jul 28 10:07:43.460417 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.460051    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab7f977a-7d91-4d97-85ca-f07a9d373ee8-etc-kubernetes\") pod \"multus-jlf2c\" (UID: \"ab7f977a-7d91-4d97-85ca-f07a9d373ee8\") " pod="openshift-multus/multus-jlf2c"
Jul 28 10:07:43.460417 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.460075    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/55104d0d-d660-4142-9b73-3277e628eefa-socket-dir\") pod \"aws-ebs-csi-driver-node-vrkpg\" (UID: \"55104d0d-d660-4142-9b73-3277e628eefa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vrkpg"
Jul 28 10:07:43.461273 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:43.460143    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 10:07:43.461273 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.460174    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42229f5e-156f-4abd-a2ee-700c9f340775-tmp-dir\") pod \"node-resolver-nxpwb\" (UID: \"42229f5e-156f-4abd-a2ee-700c9f340775\") " pod="openshift-dns/node-resolver-nxpwb"
Jul 28 10:07:43.461273 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:43.460202    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07ad2626-9f01-48ec-adb7-c5f4ab340b7a-original-pull-secret podName:07ad2626-9f01-48ec-adb7-c5f4ab340b7a nodeName:}" failed. No retries permitted until 2026-07-28 10:07:43.960171602 +0000 UTC m=+3.060678133 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/07ad2626-9f01-48ec-adb7-c5f4ab340b7a-original-pull-secret") pod "global-pull-secret-syncer-64pm8" (UID: "07ad2626-9f01-48ec-adb7-c5f4ab340b7a") : object "kube-system"/"original-pull-secret" not registered
Jul 28 10:07:43.461273 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.460204    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/55104d0d-d660-4142-9b73-3277e628eefa-socket-dir\") pod \"aws-ebs-csi-driver-node-vrkpg\" (UID: \"55104d0d-d660-4142-9b73-3277e628eefa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vrkpg"
Jul 28 10:07:43.461273 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.460247    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab7f977a-7d91-4d97-85ca-f07a9d373ee8-etc-kubernetes\") pod \"multus-jlf2c\" (UID: \"ab7f977a-7d91-4d97-85ca-f07a9d373ee8\") " pod="openshift-multus/multus-jlf2c"
Jul 28 10:07:43.461273 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.460282    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab7f977a-7d91-4d97-85ca-f07a9d373ee8-host-var-lib-kubelet\") pod \"multus-jlf2c\" (UID: \"ab7f977a-7d91-4d97-85ca-f07a9d373ee8\") " pod="openshift-multus/multus-jlf2c"
Jul 28 10:07:43.461273 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.460284    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/55104d0d-d660-4142-9b73-3277e628eefa-registration-dir\") pod \"aws-ebs-csi-driver-node-vrkpg\" (UID: \"55104d0d-d660-4142-9b73-3277e628eefa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vrkpg"
Jul 28 10:07:43.461273 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.460294    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ab7f977a-7d91-4d97-85ca-f07a9d373ee8-cni-binary-copy\") pod \"multus-jlf2c\" (UID: \"ab7f977a-7d91-4d97-85ca-f07a9d373ee8\") " pod="openshift-multus/multus-jlf2c"
Jul 28 10:07:43.461273 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.460407    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/42229f5e-156f-4abd-a2ee-700c9f340775-hosts-file\") pod \"node-resolver-nxpwb\" (UID: \"42229f5e-156f-4abd-a2ee-700c9f340775\") " pod="openshift-dns/node-resolver-nxpwb"
Jul 28 10:07:43.470091 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.470067    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h7nk\" (UniqueName: \"kubernetes.io/projected/42229f5e-156f-4abd-a2ee-700c9f340775-kube-api-access-2h7nk\") pod \"node-resolver-nxpwb\" (UID: \"42229f5e-156f-4abd-a2ee-700c9f340775\") " pod="openshift-dns/node-resolver-nxpwb"
Jul 28 10:07:43.470329 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.470195    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m48lj\" (UniqueName: \"kubernetes.io/projected/ab7f977a-7d91-4d97-85ca-f07a9d373ee8-kube-api-access-m48lj\") pod \"multus-jlf2c\" (UID: \"ab7f977a-7d91-4d97-85ca-f07a9d373ee8\") " pod="openshift-multus/multus-jlf2c"
Jul 28 10:07:43.470616 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.470594    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd8b4\" (UniqueName: \"kubernetes.io/projected/55104d0d-d660-4142-9b73-3277e628eefa-kube-api-access-kd8b4\") pod \"aws-ebs-csi-driver-node-vrkpg\" (UID: \"55104d0d-d660-4142-9b73-3277e628eefa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vrkpg"
Jul 28 10:07:43.474628 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.474572    2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 10:07:43.545531 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.545500    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5z6x8"
Jul 28 10:07:43.556506 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.556482    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-545mm"
Jul 28 10:07:43.565260 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.565168    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vvkrx"
Jul 28 10:07:43.570677 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.570660    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-v94ww"
Jul 28 10:07:43.580249 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.580230    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rrxnl"
Jul 28 10:07:43.588939 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.588920    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lqnk6"
Jul 28 10:07:43.597802 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.597782    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vrkpg"
Jul 28 10:07:43.607339 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.607315    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nxpwb"
Jul 28 10:07:43.612842 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.612824    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jlf2c"
Jul 28 10:07:43.863195 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.863094    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19db4187-c2f2-46d9-bb21-ce1d3c12c422-metrics-certs\") pod \"network-metrics-daemon-ldq5b\" (UID: \"19db4187-c2f2-46d9-bb21-ce1d3c12c422\") " pod="openshift-multus/network-metrics-daemon-ldq5b"
Jul 28 10:07:43.863347 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:43.863238    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:07:43.863347 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:43.863296    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19db4187-c2f2-46d9-bb21-ce1d3c12c422-metrics-certs podName:19db4187-c2f2-46d9-bb21-ce1d3c12c422 nodeName:}" failed. No retries permitted until 2026-07-28 10:07:44.863276842 +0000 UTC m=+3.963783576 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/19db4187-c2f2-46d9-bb21-ce1d3c12c422-metrics-certs") pod "network-metrics-daemon-ldq5b" (UID: "19db4187-c2f2-46d9-bb21-ce1d3c12c422") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:07:43.963597 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.963560    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gzlj\" (UniqueName: \"kubernetes.io/projected/d329552c-d2a3-4960-9130-bc9e211efc18-kube-api-access-2gzlj\") pod \"network-check-target-7dfzm\" (UID: \"d329552c-d2a3-4960-9130-bc9e211efc18\") " pod="openshift-network-diagnostics/network-check-target-7dfzm"
Jul 28 10:07:43.963742 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:43.963604    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/07ad2626-9f01-48ec-adb7-c5f4ab340b7a-original-pull-secret\") pod \"global-pull-secret-syncer-64pm8\" (UID: \"07ad2626-9f01-48ec-adb7-c5f4ab340b7a\") " pod="kube-system/global-pull-secret-syncer-64pm8"
Jul 28 10:07:43.963742 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:43.963705    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 10:07:43.963742 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:43.963717    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 10:07:43.963742 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:43.963738    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 10:07:43.963872 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:43.963748    2583 projected.go:196] Error preparing data for projected volume kube-api-access-2gzlj for pod openshift-network-diagnostics/network-check-target-7dfzm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:07:43.963872 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:43.963757    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07ad2626-9f01-48ec-adb7-c5f4ab340b7a-original-pull-secret podName:07ad2626-9f01-48ec-adb7-c5f4ab340b7a nodeName:}" failed. No retries permitted until 2026-07-28 10:07:44.96374094 +0000 UTC m=+4.064247487 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/07ad2626-9f01-48ec-adb7-c5f4ab340b7a-original-pull-secret") pod "global-pull-secret-syncer-64pm8" (UID: "07ad2626-9f01-48ec-adb7-c5f4ab340b7a") : object "kube-system"/"original-pull-secret" not registered
Jul 28 10:07:43.963872 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:43.963784    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d329552c-d2a3-4960-9130-bc9e211efc18-kube-api-access-2gzlj podName:d329552c-d2a3-4960-9130-bc9e211efc18 nodeName:}" failed. No retries permitted until 2026-07-28 10:07:44.963772721 +0000 UTC m=+4.064279252 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-2gzlj" (UniqueName: "kubernetes.io/projected/d329552c-d2a3-4960-9130-bc9e211efc18-kube-api-access-2gzlj") pod "network-check-target-7dfzm" (UID: "d329552c-d2a3-4960-9130-bc9e211efc18") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:07:44.266589 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:44.266559    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeccf40b_0096_40d3_b231_d7ce00688cc9.slice/crio-8d0b5974b79df5031f6dd76205c9643e90d2246b835eaeaff9e4332593e75df2 WatchSource:0}: Error finding container 8d0b5974b79df5031f6dd76205c9643e90d2246b835eaeaff9e4332593e75df2: Status 404 returned error can't find the container with id 8d0b5974b79df5031f6dd76205c9643e90d2246b835eaeaff9e4332593e75df2
Jul 28 10:07:44.280623 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:44.280600    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab7f977a_7d91_4d97_85ca_f07a9d373ee8.slice/crio-ce3a46666d9ae2c4d2afd4727aeadb7c712df041d0508a8665297ca20f50ac9a WatchSource:0}: Error finding container ce3a46666d9ae2c4d2afd4727aeadb7c712df041d0508a8665297ca20f50ac9a: Status 404 returned error can't find the container with id ce3a46666d9ae2c4d2afd4727aeadb7c712df041d0508a8665297ca20f50ac9a
Jul 28 10:07:44.281094 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:44.281007    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a872d0f_af6f_4fd4_bf8f_f763ce610da4.slice/crio-8e8ad7b949255e18cbcbc3fe222ca726376fed8c82f2247908264bb385fb9359 WatchSource:0}: Error finding container 8e8ad7b949255e18cbcbc3fe222ca726376fed8c82f2247908264bb385fb9359: Status 404 returned error can't find the container with id 8e8ad7b949255e18cbcbc3fe222ca726376fed8c82f2247908264bb385fb9359
Jul 28 10:07:44.284202 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:44.284166    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod675b6659_b757_4eca_ba86_28b21af90501.slice/crio-6ef5722057d706c027b2f898eb2a2e3c061ed13edf9cafe3f755fa8250d14844 WatchSource:0}: Error finding container 6ef5722057d706c027b2f898eb2a2e3c061ed13edf9cafe3f755fa8250d14844: Status 404 returned error can't find the container with id 6ef5722057d706c027b2f898eb2a2e3c061ed13edf9cafe3f755fa8250d14844
Jul 28 10:07:44.286136 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:44.286103    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca02c39d_85bd_4cea_ab02_19f67555f81a.slice/crio-8a339119c23e8c60506d99eaf32c650a6ae5ce8fbd049883cf3673ba04b11968 WatchSource:0}: Error finding container 8a339119c23e8c60506d99eaf32c650a6ae5ce8fbd049883cf3673ba04b11968: Status 404 returned error can't find the container with id 8a339119c23e8c60506d99eaf32c650a6ae5ce8fbd049883cf3673ba04b11968
Jul 28 10:07:44.286880 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:44.286861    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8490536f_2f40_4389_bf3f_d5cab1ab3a00.slice/crio-afb51f7ac89a081e9076d5c36658930cf4f2cf69aa33107610c6fa09ae5b94a4 WatchSource:0}: Error finding container afb51f7ac89a081e9076d5c36658930cf4f2cf69aa33107610c6fa09ae5b94a4: Status 404 returned error can't find the container with id afb51f7ac89a081e9076d5c36658930cf4f2cf69aa33107610c6fa09ae5b94a4
Jul 28 10:07:44.288626 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:44.288312    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55104d0d_d660_4142_9b73_3277e628eefa.slice/crio-81636c7c634f1c5371ab60fb79e10dec45ed61357a3696be0055c9091c3e129d WatchSource:0}: Error finding container 81636c7c634f1c5371ab60fb79e10dec45ed61357a3696be0055c9091c3e129d: Status 404 returned error can't find the container with id 81636c7c634f1c5371ab60fb79e10dec45ed61357a3696be0055c9091c3e129d
Jul 28 10:07:44.289328 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:44.289214    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17fe592b_e2cc_42ad_8803_0944d20e4dc0.slice/crio-3b08860374420b814e21a88b0ed65b378f01b922d13e6877312a7b79a61e4b55 WatchSource:0}: Error finding container 3b08860374420b814e21a88b0ed65b378f01b922d13e6877312a7b79a61e4b55: Status 404 returned error can't find the container with id 3b08860374420b814e21a88b0ed65b378f01b922d13e6877312a7b79a61e4b55
Jul 28 10:07:44.291468 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:07:44.290818    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42229f5e_156f_4abd_a2ee_700c9f340775.slice/crio-11108e2303d0cd0f086d10f820ed39a58afdd49b93376a00983b1e9ea26b0fba WatchSource:0}: Error finding container 11108e2303d0cd0f086d10f820ed39a58afdd49b93376a00983b1e9ea26b0fba: Status 404 returned error can't find the container with id 11108e2303d0cd0f086d10f820ed39a58afdd49b93376a00983b1e9ea26b0fba
Jul 28 10:07:44.291468 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:44.290889    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 10:02:43 +0000 UTC" deadline="2028-01-16 00:44:10.452719566 +0000 UTC"
Jul 28 10:07:44.291468 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:44.290930    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12878h36m26.161793205s"
Jul 28 10:07:44.433429 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:44.433276    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldq5b"
Jul 28 10:07:44.433559 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:44.433539    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldq5b" podUID="19db4187-c2f2-46d9-bb21-ce1d3c12c422"
Jul 28 10:07:44.441745 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:44.441705    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vvkrx" event={"ID":"17fe592b-e2cc-42ad-8803-0944d20e4dc0","Type":"ContainerStarted","Data":"3b08860374420b814e21a88b0ed65b378f01b922d13e6877312a7b79a61e4b55"}
Jul 28 10:07:44.442835 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:44.442813    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vrkpg" event={"ID":"55104d0d-d660-4142-9b73-3277e628eefa","Type":"ContainerStarted","Data":"81636c7c634f1c5371ab60fb79e10dec45ed61357a3696be0055c9091c3e129d"}
Jul 28 10:07:44.443775 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:44.443754    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-545mm" event={"ID":"675b6659-b757-4eca-ba86-28b21af90501","Type":"ContainerStarted","Data":"6ef5722057d706c027b2f898eb2a2e3c061ed13edf9cafe3f755fa8250d14844"}
Jul 28 10:07:44.444750 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:44.444732    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5z6x8" event={"ID":"1a872d0f-af6f-4fd4-bf8f-f763ce610da4","Type":"ContainerStarted","Data":"8e8ad7b949255e18cbcbc3fe222ca726376fed8c82f2247908264bb385fb9359"}
Jul 28 10:07:44.445791 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:44.445774    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jlf2c" event={"ID":"ab7f977a-7d91-4d97-85ca-f07a9d373ee8","Type":"ContainerStarted","Data":"ce3a46666d9ae2c4d2afd4727aeadb7c712df041d0508a8665297ca20f50ac9a"}
Jul 28 10:07:44.447774 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:44.447755    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-110.ec2.internal" event={"ID":"c2a2b445e0df91dcaba3ba620484e779","Type":"ContainerStarted","Data":"31e97d89f342871e7c4d3ba001b2c7164436ff8c0b210069f426511a2929e615"}
Jul 28 10:07:44.448984 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:44.448922    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nxpwb" event={"ID":"42229f5e-156f-4abd-a2ee-700c9f340775","Type":"ContainerStarted","Data":"11108e2303d0cd0f086d10f820ed39a58afdd49b93376a00983b1e9ea26b0fba"}
Jul 28 10:07:44.450117 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:44.450095    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lqnk6" event={"ID":"8490536f-2f40-4389-bf3f-d5cab1ab3a00","Type":"ContainerStarted","Data":"afb51f7ac89a081e9076d5c36658930cf4f2cf69aa33107610c6fa09ae5b94a4"}
Jul 28 10:07:44.451274 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:44.451258    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rrxnl" event={"ID":"ca02c39d-85bd-4cea-ab02-19f67555f81a","Type":"ContainerStarted","Data":"8a339119c23e8c60506d99eaf32c650a6ae5ce8fbd049883cf3673ba04b11968"}
Jul 28 10:07:44.452222 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:44.452205    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-v94ww" event={"ID":"eeccf40b-0096-40d3-b231-d7ce00688cc9","Type":"ContainerStarted","Data":"8d0b5974b79df5031f6dd76205c9643e90d2246b835eaeaff9e4332593e75df2"}
Jul 28 10:07:44.870573 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:44.870490    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19db4187-c2f2-46d9-bb21-ce1d3c12c422-metrics-certs\") pod \"network-metrics-daemon-ldq5b\" (UID: \"19db4187-c2f2-46d9-bb21-ce1d3c12c422\") " pod="openshift-multus/network-metrics-daemon-ldq5b"
Jul 28 10:07:44.870767 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:44.870652    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:07:44.870767 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:44.870720    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19db4187-c2f2-46d9-bb21-ce1d3c12c422-metrics-certs podName:19db4187-c2f2-46d9-bb21-ce1d3c12c422 nodeName:}" failed. No retries permitted until 2026-07-28 10:07:46.870701618 +0000 UTC m=+5.971208164 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/19db4187-c2f2-46d9-bb21-ce1d3c12c422-metrics-certs") pod "network-metrics-daemon-ldq5b" (UID: "19db4187-c2f2-46d9-bb21-ce1d3c12c422") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:07:44.971251 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:44.971214    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gzlj\" (UniqueName: \"kubernetes.io/projected/d329552c-d2a3-4960-9130-bc9e211efc18-kube-api-access-2gzlj\") pod \"network-check-target-7dfzm\" (UID: \"d329552c-d2a3-4960-9130-bc9e211efc18\") " pod="openshift-network-diagnostics/network-check-target-7dfzm"
Jul 28 10:07:44.971450 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:44.971269    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/07ad2626-9f01-48ec-adb7-c5f4ab340b7a-original-pull-secret\") pod \"global-pull-secret-syncer-64pm8\" (UID: \"07ad2626-9f01-48ec-adb7-c5f4ab340b7a\") " pod="kube-system/global-pull-secret-syncer-64pm8"
Jul 28 10:07:44.971450 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:44.971422    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 10:07:44.971567 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:44.971484    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07ad2626-9f01-48ec-adb7-c5f4ab340b7a-original-pull-secret podName:07ad2626-9f01-48ec-adb7-c5f4ab340b7a nodeName:}" failed. No retries permitted until 2026-07-28 10:07:46.971466446 +0000 UTC m=+6.071972982 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/07ad2626-9f01-48ec-adb7-c5f4ab340b7a-original-pull-secret") pod "global-pull-secret-syncer-64pm8" (UID: "07ad2626-9f01-48ec-adb7-c5f4ab340b7a") : object "kube-system"/"original-pull-secret" not registered
Jul 28 10:07:44.971892 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:44.971877    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 10:07:44.971952 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:44.971898    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 10:07:44.971952 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:44.971911    2583 projected.go:196] Error preparing data for projected volume kube-api-access-2gzlj for pod openshift-network-diagnostics/network-check-target-7dfzm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:07:44.972051 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:44.971957    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d329552c-d2a3-4960-9130-bc9e211efc18-kube-api-access-2gzlj podName:d329552c-d2a3-4960-9130-bc9e211efc18 nodeName:}" failed. No retries permitted until 2026-07-28 10:07:46.971941749 +0000 UTC m=+6.072448298 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-2gzlj" (UniqueName: "kubernetes.io/projected/d329552c-d2a3-4960-9130-bc9e211efc18-kube-api-access-2gzlj") pod "network-check-target-7dfzm" (UID: "d329552c-d2a3-4960-9130-bc9e211efc18") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:07:45.291946 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:45.291830    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 10:02:43 +0000 UTC" deadline="2028-03-28 10:14:46.804635284 +0000 UTC"
Jul 28 10:07:45.291946 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:45.291868    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14616h7m1.512772054s"
Jul 28 10:07:45.434321 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:45.434293    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7dfzm"
Jul 28 10:07:45.434487 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:45.434411    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7dfzm" podUID="d329552c-d2a3-4960-9130-bc9e211efc18"
Jul 28 10:07:45.436365 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:45.436338    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-64pm8"
Jul 28 10:07:45.436500 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:45.436477    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-64pm8" podUID="07ad2626-9f01-48ec-adb7-c5f4ab340b7a"
Jul 28 10:07:45.466702 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:45.466668    2583 generic.go:356] "Generic (PLEG): container finished" podID="2b84527c672e7fff4b21007a2e0dc2c6" containerID="f2be94164ba72a107c08e9645eae809004ccb4327be0ad2e3c260a87a2566e34" exitCode=0
Jul 28 10:07:45.466834 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:45.466748    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-110.ec2.internal" event={"ID":"2b84527c672e7fff4b21007a2e0dc2c6","Type":"ContainerDied","Data":"f2be94164ba72a107c08e9645eae809004ccb4327be0ad2e3c260a87a2566e34"}
Jul 28 10:07:45.486011 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:45.485960    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-110.ec2.internal" podStartSLOduration=3.485940973 podStartE2EDuration="3.485940973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:07:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:07:44.461877366 +0000 UTC m=+3.562383930" watchObservedRunningTime="2026-07-28 10:07:45.485940973 +0000 UTC m=+4.586447531"
Jul 28 10:07:46.432496 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:46.432458    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldq5b"
Jul 28 10:07:46.432973 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:46.432613    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldq5b" podUID="19db4187-c2f2-46d9-bb21-ce1d3c12c422"
Jul 28 10:07:46.482591 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:46.482275    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-110.ec2.internal" event={"ID":"2b84527c672e7fff4b21007a2e0dc2c6","Type":"ContainerStarted","Data":"b64679b760c29df9999e68a88f4fbe8ad0965e1f21d90e43c79ef14def0580d8"}
Jul 28 10:07:46.500939 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:46.500886    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-110.ec2.internal" podStartSLOduration=4.500869062 podStartE2EDuration="4.500869062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:07:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:07:46.49931728 +0000 UTC m=+5.599823836" watchObservedRunningTime="2026-07-28 10:07:46.500869062 +0000 UTC m=+5.601375617"
Jul 28 10:07:46.885985 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:46.885896    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19db4187-c2f2-46d9-bb21-ce1d3c12c422-metrics-certs\") pod \"network-metrics-daemon-ldq5b\" (UID: \"19db4187-c2f2-46d9-bb21-ce1d3c12c422\") " pod="openshift-multus/network-metrics-daemon-ldq5b"
Jul 28 10:07:46.886144 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:46.886049    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:07:46.886144 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:46.886108    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19db4187-c2f2-46d9-bb21-ce1d3c12c422-metrics-certs podName:19db4187-c2f2-46d9-bb21-ce1d3c12c422 nodeName:}" failed. No retries permitted until 2026-07-28 10:07:50.886089036 +0000 UTC m=+9.986595582 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/19db4187-c2f2-46d9-bb21-ce1d3c12c422-metrics-certs") pod "network-metrics-daemon-ldq5b" (UID: "19db4187-c2f2-46d9-bb21-ce1d3c12c422") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:07:46.987439 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:46.987152    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gzlj\" (UniqueName: \"kubernetes.io/projected/d329552c-d2a3-4960-9130-bc9e211efc18-kube-api-access-2gzlj\") pod \"network-check-target-7dfzm\" (UID: \"d329552c-d2a3-4960-9130-bc9e211efc18\") " pod="openshift-network-diagnostics/network-check-target-7dfzm"
Jul 28 10:07:46.987439 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:46.987260    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/07ad2626-9f01-48ec-adb7-c5f4ab340b7a-original-pull-secret\") pod \"global-pull-secret-syncer-64pm8\" (UID: \"07ad2626-9f01-48ec-adb7-c5f4ab340b7a\") " pod="kube-system/global-pull-secret-syncer-64pm8"
Jul 28 10:07:46.987439 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:46.987324    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 10:07:46.987439 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:46.987354    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 10:07:46.987439 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:46.987368    2583 projected.go:196] Error preparing data for projected volume kube-api-access-2gzlj for pod openshift-network-diagnostics/network-check-target-7dfzm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:07:46.987439 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:46.987382    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 10:07:46.987439 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:46.987436    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d329552c-d2a3-4960-9130-bc9e211efc18-kube-api-access-2gzlj podName:d329552c-d2a3-4960-9130-bc9e211efc18 nodeName:}" failed. No retries permitted until 2026-07-28 10:07:50.987406416 +0000 UTC m=+10.087912963 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-2gzlj" (UniqueName: "kubernetes.io/projected/d329552c-d2a3-4960-9130-bc9e211efc18-kube-api-access-2gzlj") pod "network-check-target-7dfzm" (UID: "d329552c-d2a3-4960-9130-bc9e211efc18") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:07:46.987946 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:46.987458    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07ad2626-9f01-48ec-adb7-c5f4ab340b7a-original-pull-secret podName:07ad2626-9f01-48ec-adb7-c5f4ab340b7a nodeName:}" failed. No retries permitted until 2026-07-28 10:07:50.9874483 +0000 UTC m=+10.087954860 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/07ad2626-9f01-48ec-adb7-c5f4ab340b7a-original-pull-secret") pod "global-pull-secret-syncer-64pm8" (UID: "07ad2626-9f01-48ec-adb7-c5f4ab340b7a") : object "kube-system"/"original-pull-secret" not registered
Jul 28 10:07:47.434320 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:47.434290    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7dfzm"
Jul 28 10:07:47.434878 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:47.434421    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7dfzm" podUID="d329552c-d2a3-4960-9130-bc9e211efc18"
Jul 28 10:07:47.434956 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:47.434891    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-64pm8"
Jul 28 10:07:47.435071 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:47.434993    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-64pm8" podUID="07ad2626-9f01-48ec-adb7-c5f4ab340b7a"
Jul 28 10:07:48.433072 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:48.433032    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldq5b"
Jul 28 10:07:48.433383 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:48.433195    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldq5b" podUID="19db4187-c2f2-46d9-bb21-ce1d3c12c422"
Jul 28 10:07:49.432540 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:49.432510    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-64pm8"
Jul 28 10:07:49.432994 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:49.432511    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7dfzm"
Jul 28 10:07:49.432994 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:49.432645    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-64pm8" podUID="07ad2626-9f01-48ec-adb7-c5f4ab340b7a"
Jul 28 10:07:49.432994 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:49.432689    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7dfzm" podUID="d329552c-d2a3-4960-9130-bc9e211efc18"
Jul 28 10:07:50.432799 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:50.432768    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldq5b"
Jul 28 10:07:50.433258 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:50.432884    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldq5b" podUID="19db4187-c2f2-46d9-bb21-ce1d3c12c422"
Jul 28 10:07:50.919354 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:50.919210    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19db4187-c2f2-46d9-bb21-ce1d3c12c422-metrics-certs\") pod \"network-metrics-daemon-ldq5b\" (UID: \"19db4187-c2f2-46d9-bb21-ce1d3c12c422\") " pod="openshift-multus/network-metrics-daemon-ldq5b"
Jul 28 10:07:50.919354 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:50.919350    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:07:50.919560 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:50.919418    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19db4187-c2f2-46d9-bb21-ce1d3c12c422-metrics-certs podName:19db4187-c2f2-46d9-bb21-ce1d3c12c422 nodeName:}" failed. No retries permitted until 2026-07-28 10:07:58.919399855 +0000 UTC m=+18.019906387 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/19db4187-c2f2-46d9-bb21-ce1d3c12c422-metrics-certs") pod "network-metrics-daemon-ldq5b" (UID: "19db4187-c2f2-46d9-bb21-ce1d3c12c422") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:07:51.020444 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:51.020403    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/07ad2626-9f01-48ec-adb7-c5f4ab340b7a-original-pull-secret\") pod \"global-pull-secret-syncer-64pm8\" (UID: \"07ad2626-9f01-48ec-adb7-c5f4ab340b7a\") " pod="kube-system/global-pull-secret-syncer-64pm8"
Jul 28 10:07:51.020601 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:51.020516    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gzlj\" (UniqueName: \"kubernetes.io/projected/d329552c-d2a3-4960-9130-bc9e211efc18-kube-api-access-2gzlj\") pod \"network-check-target-7dfzm\" (UID: \"d329552c-d2a3-4960-9130-bc9e211efc18\") " pod="openshift-network-diagnostics/network-check-target-7dfzm"
Jul 28 10:07:51.020678 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:51.020603    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 10:07:51.020678 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:51.020656    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 10:07:51.020678 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:51.020674    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07ad2626-9f01-48ec-adb7-c5f4ab340b7a-original-pull-secret podName:07ad2626-9f01-48ec-adb7-c5f4ab340b7a nodeName:}" failed. No retries permitted until 2026-07-28 10:07:59.020654528 +0000 UTC m=+18.121161074 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/07ad2626-9f01-48ec-adb7-c5f4ab340b7a-original-pull-secret") pod "global-pull-secret-syncer-64pm8" (UID: "07ad2626-9f01-48ec-adb7-c5f4ab340b7a") : object "kube-system"/"original-pull-secret" not registered
Jul 28 10:07:51.020827 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:51.020675    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 10:07:51.020827 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:51.020697    2583 projected.go:196] Error preparing data for projected volume kube-api-access-2gzlj for pod openshift-network-diagnostics/network-check-target-7dfzm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:07:51.020827 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:51.020736    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d329552c-d2a3-4960-9130-bc9e211efc18-kube-api-access-2gzlj podName:d329552c-d2a3-4960-9130-bc9e211efc18 nodeName:}" failed. No retries permitted until 2026-07-28 10:07:59.020725461 +0000 UTC m=+18.121231997 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-2gzlj" (UniqueName: "kubernetes.io/projected/d329552c-d2a3-4960-9130-bc9e211efc18-kube-api-access-2gzlj") pod "network-check-target-7dfzm" (UID: "d329552c-d2a3-4960-9130-bc9e211efc18") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:07:51.433885 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:51.433852    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-64pm8"
Jul 28 10:07:51.434396 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:51.433968    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-64pm8" podUID="07ad2626-9f01-48ec-adb7-c5f4ab340b7a"
Jul 28 10:07:51.434396 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:51.434378    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7dfzm"
Jul 28 10:07:51.434512 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:51.434471    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7dfzm" podUID="d329552c-d2a3-4960-9130-bc9e211efc18"
Jul 28 10:07:52.433048 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:52.432965    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldq5b"
Jul 28 10:07:52.433234 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:52.433112    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldq5b" podUID="19db4187-c2f2-46d9-bb21-ce1d3c12c422"
Jul 28 10:07:53.433323 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:53.433276    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-64pm8"
Jul 28 10:07:53.433814 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:53.433272    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7dfzm"
Jul 28 10:07:53.433814 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:53.433418    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-64pm8" podUID="07ad2626-9f01-48ec-adb7-c5f4ab340b7a"
Jul 28 10:07:53.433814 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:53.433501    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7dfzm" podUID="d329552c-d2a3-4960-9130-bc9e211efc18"
Jul 28 10:07:54.432516 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:54.432485    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldq5b"
Jul 28 10:07:54.432676 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:54.432588    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldq5b" podUID="19db4187-c2f2-46d9-bb21-ce1d3c12c422"
Jul 28 10:07:55.433080 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:55.433048    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-64pm8"
Jul 28 10:07:55.433503 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:55.433192    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-64pm8" podUID="07ad2626-9f01-48ec-adb7-c5f4ab340b7a"
Jul 28 10:07:55.433503 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:55.433231    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7dfzm"
Jul 28 10:07:55.433503 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:55.433343    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7dfzm" podUID="d329552c-d2a3-4960-9130-bc9e211efc18"
Jul 28 10:07:56.432965 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:56.432937    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldq5b"
Jul 28 10:07:56.433128 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:56.433050    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldq5b" podUID="19db4187-c2f2-46d9-bb21-ce1d3c12c422"
Jul 28 10:07:57.433059 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:57.433027    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7dfzm"
Jul 28 10:07:57.433228 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:57.433145    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7dfzm" podUID="d329552c-d2a3-4960-9130-bc9e211efc18"
Jul 28 10:07:57.433228 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:57.433200    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-64pm8"
Jul 28 10:07:57.433520 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:57.433289    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-64pm8" podUID="07ad2626-9f01-48ec-adb7-c5f4ab340b7a"
Jul 28 10:07:58.433124 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:58.433088    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldq5b"
Jul 28 10:07:58.433368 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:58.433251    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldq5b" podUID="19db4187-c2f2-46d9-bb21-ce1d3c12c422"
Jul 28 10:07:58.983008 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:58.982976    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19db4187-c2f2-46d9-bb21-ce1d3c12c422-metrics-certs\") pod \"network-metrics-daemon-ldq5b\" (UID: \"19db4187-c2f2-46d9-bb21-ce1d3c12c422\") " pod="openshift-multus/network-metrics-daemon-ldq5b"
Jul 28 10:07:58.983206 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:58.983107    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:07:58.983206 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:58.983175    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19db4187-c2f2-46d9-bb21-ce1d3c12c422-metrics-certs podName:19db4187-c2f2-46d9-bb21-ce1d3c12c422 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:14.983155811 +0000 UTC m=+34.083662357 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/19db4187-c2f2-46d9-bb21-ce1d3c12c422-metrics-certs") pod "network-metrics-daemon-ldq5b" (UID: "19db4187-c2f2-46d9-bb21-ce1d3c12c422") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:07:59.083912 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:59.083840    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gzlj\" (UniqueName: \"kubernetes.io/projected/d329552c-d2a3-4960-9130-bc9e211efc18-kube-api-access-2gzlj\") pod \"network-check-target-7dfzm\" (UID: \"d329552c-d2a3-4960-9130-bc9e211efc18\") " pod="openshift-network-diagnostics/network-check-target-7dfzm"
Jul 28 10:07:59.083912 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:59.083878    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/07ad2626-9f01-48ec-adb7-c5f4ab340b7a-original-pull-secret\") pod \"global-pull-secret-syncer-64pm8\" (UID: \"07ad2626-9f01-48ec-adb7-c5f4ab340b7a\") " pod="kube-system/global-pull-secret-syncer-64pm8"
Jul 28 10:07:59.084129 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:59.084002    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 10:07:59.084129 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:59.084006    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 10:07:59.084129 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:59.084028    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 10:07:59.084129 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:59.084038    2583 projected.go:196] Error preparing data for projected volume kube-api-access-2gzlj for pod openshift-network-diagnostics/network-check-target-7dfzm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:07:59.084129 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:59.084052    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07ad2626-9f01-48ec-adb7-c5f4ab340b7a-original-pull-secret podName:07ad2626-9f01-48ec-adb7-c5f4ab340b7a nodeName:}" failed. No retries permitted until 2026-07-28 10:08:15.084037347 +0000 UTC m=+34.184543885 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/07ad2626-9f01-48ec-adb7-c5f4ab340b7a-original-pull-secret") pod "global-pull-secret-syncer-64pm8" (UID: "07ad2626-9f01-48ec-adb7-c5f4ab340b7a") : object "kube-system"/"original-pull-secret" not registered
Jul 28 10:07:59.084129 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:59.084089    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d329552c-d2a3-4960-9130-bc9e211efc18-kube-api-access-2gzlj podName:d329552c-d2a3-4960-9130-bc9e211efc18 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:15.084071551 +0000 UTC m=+34.184578087 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-2gzlj" (UniqueName: "kubernetes.io/projected/d329552c-d2a3-4960-9130-bc9e211efc18-kube-api-access-2gzlj") pod "network-check-target-7dfzm" (UID: "d329552c-d2a3-4960-9130-bc9e211efc18") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:07:59.432570 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:59.432486    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7dfzm"
Jul 28 10:07:59.432730 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:07:59.432495    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-64pm8"
Jul 28 10:07:59.432730 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:59.432609    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7dfzm" podUID="d329552c-d2a3-4960-9130-bc9e211efc18"
Jul 28 10:07:59.432730 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:07:59.432687    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-64pm8" podUID="07ad2626-9f01-48ec-adb7-c5f4ab340b7a"
Jul 28 10:08:00.433247 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:00.433211    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldq5b"
Jul 28 10:08:00.433740 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:00.433366    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldq5b" podUID="19db4187-c2f2-46d9-bb21-ce1d3c12c422"
Jul 28 10:08:01.434065 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:01.434028    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7dfzm"
Jul 28 10:08:01.434449 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:01.434122    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7dfzm" podUID="d329552c-d2a3-4960-9130-bc9e211efc18"
Jul 28 10:08:01.434449 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:01.434218    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-64pm8"
Jul 28 10:08:01.434449 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:01.434328    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-64pm8" podUID="07ad2626-9f01-48ec-adb7-c5f4ab340b7a"
Jul 28 10:08:02.433332 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:02.433295    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldq5b"
Jul 28 10:08:02.433487 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:02.433404    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldq5b" podUID="19db4187-c2f2-46d9-bb21-ce1d3c12c422"
Jul 28 10:08:03.432967 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:03.432567    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7dfzm"
Jul 28 10:08:03.433555 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:03.432603    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-64pm8"
Jul 28 10:08:03.433555 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:03.433076    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7dfzm" podUID="d329552c-d2a3-4960-9130-bc9e211efc18"
Jul 28 10:08:03.433555 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:03.433114    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-64pm8" podUID="07ad2626-9f01-48ec-adb7-c5f4ab340b7a"
Jul 28 10:08:03.515496 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:03.515467    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jlf2c" event={"ID":"ab7f977a-7d91-4d97-85ca-f07a9d373ee8","Type":"ContainerStarted","Data":"e10d3adc33af0be7abab7128fb95c8fa5f5b0835a1d84855e79c6e355a5d084f"}
Jul 28 10:08:03.517240 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:03.517215    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nxpwb" event={"ID":"42229f5e-156f-4abd-a2ee-700c9f340775","Type":"ContainerStarted","Data":"3de3ba2a4f949a455fd23aff5f00a5d1dfb5993a624ef484a652b12115ef6f4d"}
Jul 28 10:08:03.519080 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:03.518833    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lqnk6" event={"ID":"8490536f-2f40-4389-bf3f-d5cab1ab3a00","Type":"ContainerStarted","Data":"71ad7a3797c3982ead0922fed9f6047461890c3e44cc0b62cd8bcc78690bb626"}
Jul 28 10:08:03.522033 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:03.522004    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rrxnl" event={"ID":"ca02c39d-85bd-4cea-ab02-19f67555f81a","Type":"ContainerStarted","Data":"4951cd85baf6f245a4ef64a43e82a20d447dc313a7eebc625b3e624237b7166c"}
Jul 28 10:08:03.522135 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:03.522075    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rrxnl" event={"ID":"ca02c39d-85bd-4cea-ab02-19f67555f81a","Type":"ContainerStarted","Data":"502b68bc239f4de896e4179b49fcf896558514f4362223d62a5d8d36cf7817ff"}
Jul 28 10:08:03.522135 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:03.522106    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rrxnl" event={"ID":"ca02c39d-85bd-4cea-ab02-19f67555f81a","Type":"ContainerStarted","Data":"4c3db73e6ec370ff967a0046980a6d15d6a9361d0f1add1f6c4fc02d5efbf5f7"}
Jul 28 10:08:03.522135 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:03.522134    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rrxnl" event={"ID":"ca02c39d-85bd-4cea-ab02-19f67555f81a","Type":"ContainerStarted","Data":"29a614720b6ac7877efa068f1d9a4d65a94bc539279a450e3818806464532e54"}
Jul 28 10:08:03.522309 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:03.522172    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rrxnl" event={"ID":"ca02c39d-85bd-4cea-ab02-19f67555f81a","Type":"ContainerStarted","Data":"890e010df88ff92a6681bd26ffc477bc343370e9e89ca5ca85b027f96f029b94"}
Jul 28 10:08:03.523961 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:03.523940    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-v94ww" event={"ID":"eeccf40b-0096-40d3-b231-d7ce00688cc9","Type":"ContainerStarted","Data":"b84702ab5af20fe5ca18b025f1aa3ec61a8bc0636874428e183baf5fa66bb82d"}
Jul 28 10:08:03.525566 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:03.525536    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vvkrx" event={"ID":"17fe592b-e2cc-42ad-8803-0944d20e4dc0","Type":"ContainerStarted","Data":"c64d0eee52b56c98a266a4a94ef77a1bd43684d0fc4f939fd9d464d18e2576e0"}
Jul 28 10:08:03.527156 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:03.527134    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vrkpg" event={"ID":"55104d0d-d660-4142-9b73-3277e628eefa","Type":"ContainerStarted","Data":"e5d8ffec5e8d7306c21eac458fd86af1175066d427fdfdd854bd5f28d3187db3"}
Jul 28 10:08:03.529106 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:03.529081    2583 generic.go:356] "Generic (PLEG): container finished" podID="675b6659-b757-4eca-ba86-28b21af90501" containerID="8e42102a86a90e66f744e8095000c9e56a2d5f016c0014276043d10199a55562" exitCode=0
Jul 28 10:08:03.529219 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:03.529115    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-545mm" event={"ID":"675b6659-b757-4eca-ba86-28b21af90501","Type":"ContainerDied","Data":"8e42102a86a90e66f744e8095000c9e56a2d5f016c0014276043d10199a55562"}
Jul 28 10:08:03.543922 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:03.543880    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-jlf2c" podStartSLOduration=3.921734384 podStartE2EDuration="22.543864412s" totalImagesPullingTime="18.622130028s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:07:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:03.543296037 +0000 UTC m=+22.643802590" watchObservedRunningTime="2026-07-28 10:08:03.543864412 +0000 UTC m=+22.644370968"
Jul 28 10:08:03.582534 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:03.582498    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-v94ww" podStartSLOduration=4.011499642 podStartE2EDuration="22.5824834s" totalImagesPullingTime="18.570983758s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:07:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:03.582319136 +0000 UTC m=+22.682825709" watchObservedRunningTime="2026-07-28 10:08:03.5824834 +0000 UTC m=+22.682989955"
Jul 28 10:08:03.595809 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:03.595772    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-lqnk6" podStartSLOduration=8.100027634 podStartE2EDuration="22.595763154s" totalImagesPullingTime="14.49573552s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:07:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:03.595236361 +0000 UTC m=+22.695742941" watchObservedRunningTime="2026-07-28 10:08:03.595763154 +0000 UTC m=+22.696269707"
Jul 28 10:08:03.612129 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:03.612093    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-nxpwb" podStartSLOduration=8.119711862 podStartE2EDuration="22.612079908s" totalImagesPullingTime="14.492368046s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:07:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:03.611606575 +0000 UTC m=+22.712113129" watchObservedRunningTime="2026-07-28 10:08:03.612079908 +0000 UTC m=+22.712586462"
Jul 28 10:08:03.630022 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:03.629956    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vvkrx" podStartSLOduration=4.055751131 podStartE2EDuration="22.629943333s" totalImagesPullingTime="18.574192202s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:07:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:03.629693394 +0000 UTC m=+22.730199969" watchObservedRunningTime="2026-07-28 10:08:03.629943333 +0000 UTC m=+22.730449887"
Jul 28 10:08:04.303812 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:04.303780    2583 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 28 10:08:04.307789 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:04.307714    2583 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T10:08:04.303802091Z","UUID":"4525d2e2-9a76-4684-b0ef-b15e07d0142e","Handler":null,"Name":"","Endpoint":""}
Jul 28 10:08:04.308885 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:04.308871    2583 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 28 10:08:04.308967 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:04.308901    2583 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 28 10:08:04.433427 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:04.433370    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldq5b"
Jul 28 10:08:04.433728 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:04.433466    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldq5b" podUID="19db4187-c2f2-46d9-bb21-ce1d3c12c422"
Jul 28 10:08:04.533719 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:04.533693    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rrxnl" event={"ID":"ca02c39d-85bd-4cea-ab02-19f67555f81a","Type":"ContainerStarted","Data":"5edd0471632da240b13d1e651cb6e1551f6df5a0e17b6c0473fe728a06a8dbff"}
Jul 28 10:08:04.535230 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:04.535206    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vrkpg" event={"ID":"55104d0d-d660-4142-9b73-3277e628eefa","Type":"ContainerStarted","Data":"3df4fe05efdfa2ceaaa46fb16ca1b6b3083357393de23f4c2af378cac4a52634"}
Jul 28 10:08:04.536696 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:04.536668    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5z6x8" event={"ID":"1a872d0f-af6f-4fd4-bf8f-f763ce610da4","Type":"ContainerStarted","Data":"0aae33fd74d29798e1d67bc0791ae16f7c4fc3a9f1f1c16174fe83d75ce2a4e9"}
Jul 28 10:08:04.552067 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:04.552026    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-5z6x8" podStartSLOduration=9.050847838 podStartE2EDuration="23.552013735s" totalImagesPullingTime="14.501165897s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:07:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:04.55158588 +0000 UTC m=+23.652092434" watchObservedRunningTime="2026-07-28 10:08:04.552013735 +0000 UTC m=+23.652520288"
Jul 28 10:08:05.433028 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:05.432996    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-64pm8"
Jul 28 10:08:05.433246 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:05.433124    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-64pm8" podUID="07ad2626-9f01-48ec-adb7-c5f4ab340b7a"
Jul 28 10:08:05.433246 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:05.433158    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7dfzm"
Jul 28 10:08:05.433344 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:05.433264    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7dfzm" podUID="d329552c-d2a3-4960-9130-bc9e211efc18"
Jul 28 10:08:06.432678 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:06.432494    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldq5b"
Jul 28 10:08:06.433128 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:06.432800    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldq5b" podUID="19db4187-c2f2-46d9-bb21-ce1d3c12c422"
Jul 28 10:08:06.542026 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:06.541989    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vrkpg" event={"ID":"55104d0d-d660-4142-9b73-3277e628eefa","Type":"ContainerStarted","Data":"b494c5b692a10642ebb373033e6292ba8c8ec218f8eb1c256325b45e4e80b5f3"}
Jul 28 10:08:06.545301 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:06.545267    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rrxnl" event={"ID":"ca02c39d-85bd-4cea-ab02-19f67555f81a","Type":"ContainerStarted","Data":"fb8029ecb6ac48f5f4dc4b4e71fd8570594fba929b30a8d68d48525cc65b22f9"}
Jul 28 10:08:06.560487 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:06.560444    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vrkpg" podStartSLOduration=4.747794501 podStartE2EDuration="25.560430162s" totalImagesPullingTime="20.812635661s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:07:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:06.560118814 +0000 UTC m=+25.660625368" watchObservedRunningTime="2026-07-28 10:08:06.560430162 +0000 UTC m=+25.660936716"
Jul 28 10:08:07.433221 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:07.433174    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7dfzm"
Jul 28 10:08:07.433661 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:07.433228    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-64pm8"
Jul 28 10:08:07.433661 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:07.433362    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7dfzm" podUID="d329552c-d2a3-4960-9130-bc9e211efc18"
Jul 28 10:08:07.433661 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:07.433462    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-64pm8" podUID="07ad2626-9f01-48ec-adb7-c5f4ab340b7a"
Jul 28 10:08:08.137432 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:08.137367    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-lqnk6"
Jul 28 10:08:08.137857 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:08.137832    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-lqnk6"
Jul 28 10:08:08.432686 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:08.432553    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldq5b"
Jul 28 10:08:08.432823 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:08.432801    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldq5b" podUID="19db4187-c2f2-46d9-bb21-ce1d3c12c422"
Jul 28 10:08:08.551250 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:08.551216    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-545mm" event={"ID":"675b6659-b757-4eca-ba86-28b21af90501","Type":"ContainerStarted","Data":"2f16c106410e6f03267d8282d0e8455fedce166bd5b64d1f36a286218bb9c26f"}
Jul 28 10:08:08.554909 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:08.554879    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rrxnl" event={"ID":"ca02c39d-85bd-4cea-ab02-19f67555f81a","Type":"ContainerStarted","Data":"b5fbb52d7d5aae0d8bb6b1d3f97906c08309f59e8d541f060f2ee364b9cbc6ec"}
Jul 28 10:08:08.555016 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:08.554967    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-lqnk6"
Jul 28 10:08:08.555427 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:08.555296    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rrxnl"
Jul 28 10:08:08.555427 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:08.555363    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rrxnl"
Jul 28 10:08:08.555579 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:08.555540    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-lqnk6"
Jul 28 10:08:08.569026 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:08.569004    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rrxnl"
Jul 28 10:08:08.614899 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:08.614856    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-rrxnl" podStartSLOduration=8.920652843 podStartE2EDuration="27.614843234s" totalImagesPullingTime="18.694190391s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:07:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:08.613665523 +0000 UTC m=+27.714172076" watchObservedRunningTime="2026-07-28 10:08:08.614843234 +0000 UTC m=+27.715349788"
Jul 28 10:08:09.432580 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:09.432553    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-64pm8"
Jul 28 10:08:09.432780 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:09.432683    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-64pm8" podUID="07ad2626-9f01-48ec-adb7-c5f4ab340b7a"
Jul 28 10:08:09.432780 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:09.432738    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7dfzm"
Jul 28 10:08:09.432880 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:09.432834    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7dfzm" podUID="d329552c-d2a3-4960-9130-bc9e211efc18"
Jul 28 10:08:09.558714 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:09.558638    2583 generic.go:356] "Generic (PLEG): container finished" podID="675b6659-b757-4eca-ba86-28b21af90501" containerID="2f16c106410e6f03267d8282d0e8455fedce166bd5b64d1f36a286218bb9c26f" exitCode=0
Jul 28 10:08:09.558714 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:09.558699    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-545mm" event={"ID":"675b6659-b757-4eca-ba86-28b21af90501","Type":"ContainerDied","Data":"2f16c106410e6f03267d8282d0e8455fedce166bd5b64d1f36a286218bb9c26f"}
Jul 28 10:08:09.559231 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:09.559214    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rrxnl"
Jul 28 10:08:09.573506 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:09.573482    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rrxnl"
Jul 28 10:08:10.432466 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:10.432439    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldq5b"
Jul 28 10:08:10.432570 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:10.432537    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldq5b" podUID="19db4187-c2f2-46d9-bb21-ce1d3c12c422"
Jul 28 10:08:10.633220 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:10.633135    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-64pm8"]
Jul 28 10:08:10.633676 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:10.633293    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-64pm8"
Jul 28 10:08:10.633676 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:10.633433    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-64pm8" podUID="07ad2626-9f01-48ec-adb7-c5f4ab340b7a"
Jul 28 10:08:10.633957 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:10.633934    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7dfzm"]
Jul 28 10:08:10.634051 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:10.634037    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7dfzm"
Jul 28 10:08:10.634148 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:10.634126    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7dfzm" podUID="d329552c-d2a3-4960-9130-bc9e211efc18"
Jul 28 10:08:10.635800 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:10.635781    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ldq5b"]
Jul 28 10:08:10.635908 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:10.635849    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldq5b"
Jul 28 10:08:10.635968 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:10.635927    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldq5b" podUID="19db4187-c2f2-46d9-bb21-ce1d3c12c422"
Jul 28 10:08:11.564968 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:11.564688    2583 generic.go:356] "Generic (PLEG): container finished" podID="675b6659-b757-4eca-ba86-28b21af90501" containerID="63ae463a2dabcc9198e768ca4452ca05e460e10228f8e9635ffc4a51d92330c9" exitCode=0
Jul 28 10:08:11.565100 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:11.564772    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-545mm" event={"ID":"675b6659-b757-4eca-ba86-28b21af90501","Type":"ContainerDied","Data":"63ae463a2dabcc9198e768ca4452ca05e460e10228f8e9635ffc4a51d92330c9"}
Jul 28 10:08:12.433108 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:12.433078    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldq5b"
Jul 28 10:08:12.433640 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:12.433081    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7dfzm"
Jul 28 10:08:12.433640 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:12.433228    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldq5b" podUID="19db4187-c2f2-46d9-bb21-ce1d3c12c422"
Jul 28 10:08:12.433640 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:12.433078    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-64pm8"
Jul 28 10:08:12.433640 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:12.433266    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7dfzm" podUID="d329552c-d2a3-4960-9130-bc9e211efc18"
Jul 28 10:08:12.433640 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:12.433342    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-64pm8" podUID="07ad2626-9f01-48ec-adb7-c5f4ab340b7a"
Jul 28 10:08:12.569648 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:12.569621    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-545mm" event={"ID":"675b6659-b757-4eca-ba86-28b21af90501","Type":"ContainerStarted","Data":"30eb973728048b78ac65ec64fd1c66118eec33a1305ad2a2861d00e2e93bef67"}
Jul 28 10:08:13.574016 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:13.573985    2583 generic.go:356] "Generic (PLEG): container finished" podID="675b6659-b757-4eca-ba86-28b21af90501" containerID="30eb973728048b78ac65ec64fd1c66118eec33a1305ad2a2861d00e2e93bef67" exitCode=0
Jul 28 10:08:13.574433 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:13.574043    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-545mm" event={"ID":"675b6659-b757-4eca-ba86-28b21af90501","Type":"ContainerDied","Data":"30eb973728048b78ac65ec64fd1c66118eec33a1305ad2a2861d00e2e93bef67"}
Jul 28 10:08:14.284637 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:14.284598    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:14.390134 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:14.390083    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:14.433270 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:14.433242    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-64pm8"
Jul 28 10:08:14.433270 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:14.433255    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldq5b"
Jul 28 10:08:14.433489 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:14.433242    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7dfzm"
Jul 28 10:08:14.433489 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:14.433364    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-64pm8" podUID="07ad2626-9f01-48ec-adb7-c5f4ab340b7a"
Jul 28 10:08:14.433489 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:14.433436    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7dfzm" podUID="d329552c-d2a3-4960-9130-bc9e211efc18"
Jul 28 10:08:14.433639 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:14.433528    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldq5b" podUID="19db4187-c2f2-46d9-bb21-ce1d3c12c422"
Jul 28 10:08:14.590233 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:14.590158    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:14.688733 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:14.688698    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:14.744345 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:14.744317    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:15.001062 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:15.000985    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19db4187-c2f2-46d9-bb21-ce1d3c12c422-metrics-certs\") pod \"network-metrics-daemon-ldq5b\" (UID: \"19db4187-c2f2-46d9-bb21-ce1d3c12c422\") " pod="openshift-multus/network-metrics-daemon-ldq5b"
Jul 28 10:08:15.001252 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:15.001120    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:08:15.001252 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:15.001196    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19db4187-c2f2-46d9-bb21-ce1d3c12c422-metrics-certs podName:19db4187-c2f2-46d9-bb21-ce1d3c12c422 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:47.001161476 +0000 UTC m=+66.101668021 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/19db4187-c2f2-46d9-bb21-ce1d3c12c422-metrics-certs") pod "network-metrics-daemon-ldq5b" (UID: "19db4187-c2f2-46d9-bb21-ce1d3c12c422") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:08:15.065423 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:15.065379    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:15.102256 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:15.102229    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gzlj\" (UniqueName: \"kubernetes.io/projected/d329552c-d2a3-4960-9130-bc9e211efc18-kube-api-access-2gzlj\") pod \"network-check-target-7dfzm\" (UID: \"d329552c-d2a3-4960-9130-bc9e211efc18\") " pod="openshift-network-diagnostics/network-check-target-7dfzm"
Jul 28 10:08:15.102405 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:15.102265    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/07ad2626-9f01-48ec-adb7-c5f4ab340b7a-original-pull-secret\") pod \"global-pull-secret-syncer-64pm8\" (UID: \"07ad2626-9f01-48ec-adb7-c5f4ab340b7a\") " pod="kube-system/global-pull-secret-syncer-64pm8"
Jul 28 10:08:15.102405 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:15.102384    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 10:08:15.102405 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:15.102387    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 10:08:15.102405 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:15.102406    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 10:08:15.102543 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:15.102416    2583 projected.go:196] Error preparing data for projected volume kube-api-access-2gzlj for pod openshift-network-diagnostics/network-check-target-7dfzm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:08:15.102543 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:15.102438    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07ad2626-9f01-48ec-adb7-c5f4ab340b7a-original-pull-secret podName:07ad2626-9f01-48ec-adb7-c5f4ab340b7a nodeName:}" failed. No retries permitted until 2026-07-28 10:08:47.102421925 +0000 UTC m=+66.202928466 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/07ad2626-9f01-48ec-adb7-c5f4ab340b7a-original-pull-secret") pod "global-pull-secret-syncer-64pm8" (UID: "07ad2626-9f01-48ec-adb7-c5f4ab340b7a") : object "kube-system"/"original-pull-secret" not registered
Jul 28 10:08:15.102543 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:15.102452    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d329552c-d2a3-4960-9130-bc9e211efc18-kube-api-access-2gzlj podName:d329552c-d2a3-4960-9130-bc9e211efc18 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:47.102446296 +0000 UTC m=+66.202952827 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-2gzlj" (UniqueName: "kubernetes.io/projected/d329552c-d2a3-4960-9130-bc9e211efc18-kube-api-access-2gzlj") pod "network-check-target-7dfzm" (UID: "d329552c-d2a3-4960-9130-bc9e211efc18") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:08:15.163723 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:15.163685    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:15.706086 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:15.706016    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-110.ec2.internal" event="NodeReady"
Jul 28 10:08:15.706465 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:15.706141    2583 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 28 10:08:15.750751 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:15.750722    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-tmddp"]
Jul 28 10:08:15.752920 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:15.752892    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tmddp"
Jul 28 10:08:15.755105 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:15.755084    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 28 10:08:15.755223 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:15.755106    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5fpr8\""
Jul 28 10:08:15.755483 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:15.755462    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 28 10:08:15.756391 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:15.756371    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-zqw4b"]
Jul 28 10:08:15.758218 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:15.758200    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zqw4b"
Jul 28 10:08:15.760050 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:15.760023    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 28 10:08:15.760267 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:15.760248    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-mbb65\""
Jul 28 10:08:15.760550 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:15.760534    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 28 10:08:15.760642 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:15.760628    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 28 10:08:15.772363 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:15.772344    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tmddp"]
Jul 28 10:08:15.773248 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:15.773229    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zqw4b"]
Jul 28 10:08:15.778660 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:15.778641    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:15.842336 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:15.842300    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:15.906304 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:15.906278    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56fcd\" (UniqueName: \"kubernetes.io/projected/be593a74-ef4a-4604-a64f-5e8ee0722fb7-kube-api-access-56fcd\") pod \"dns-default-tmddp\" (UID: \"be593a74-ef4a-4604-a64f-5e8ee0722fb7\") " pod="openshift-dns/dns-default-tmddp"
Jul 28 10:08:15.906461 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:15.906318    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/be593a74-ef4a-4604-a64f-5e8ee0722fb7-metrics-tls\") pod \"dns-default-tmddp\" (UID: \"be593a74-ef4a-4604-a64f-5e8ee0722fb7\") " pod="openshift-dns/dns-default-tmddp"
Jul 28 10:08:15.906461 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:15.906355    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2vwd\" (UniqueName: \"kubernetes.io/projected/eb3e24da-20ea-46a2-8a54-3a518d710941-kube-api-access-j2vwd\") pod \"ingress-canary-zqw4b\" (UID: \"eb3e24da-20ea-46a2-8a54-3a518d710941\") " pod="openshift-ingress-canary/ingress-canary-zqw4b"
Jul 28 10:08:15.906461 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:15.906445    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be593a74-ef4a-4604-a64f-5e8ee0722fb7-tmp-dir\") pod \"dns-default-tmddp\" (UID: \"be593a74-ef4a-4604-a64f-5e8ee0722fb7\") " pod="openshift-dns/dns-default-tmddp"
Jul 28 10:08:15.906581 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:15.906513    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/be593a74-ef4a-4604-a64f-5e8ee0722fb7-config-volume\") pod \"dns-default-tmddp\" (UID: \"be593a74-ef4a-4604-a64f-5e8ee0722fb7\") " pod="openshift-dns/dns-default-tmddp"
Jul 28 10:08:15.906581 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:15.906540    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eb3e24da-20ea-46a2-8a54-3a518d710941-cert\") pod \"ingress-canary-zqw4b\" (UID: \"eb3e24da-20ea-46a2-8a54-3a518d710941\") " pod="openshift-ingress-canary/ingress-canary-zqw4b"
Jul 28 10:08:16.007640 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:16.007558    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eb3e24da-20ea-46a2-8a54-3a518d710941-cert\") pod \"ingress-canary-zqw4b\" (UID: \"eb3e24da-20ea-46a2-8a54-3a518d710941\") " pod="openshift-ingress-canary/ingress-canary-zqw4b"
Jul 28 10:08:16.007640 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:16.007621    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56fcd\" (UniqueName: \"kubernetes.io/projected/be593a74-ef4a-4604-a64f-5e8ee0722fb7-kube-api-access-56fcd\") pod \"dns-default-tmddp\" (UID: \"be593a74-ef4a-4604-a64f-5e8ee0722fb7\") " pod="openshift-dns/dns-default-tmddp"
Jul 28 10:08:16.007844 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:16.007660    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/be593a74-ef4a-4604-a64f-5e8ee0722fb7-metrics-tls\") pod \"dns-default-tmddp\" (UID: \"be593a74-ef4a-4604-a64f-5e8ee0722fb7\") " pod="openshift-dns/dns-default-tmddp"
Jul 28 10:08:16.007844 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:16.007753    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 10:08:16.007958 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:16.007843    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be593a74-ef4a-4604-a64f-5e8ee0722fb7-metrics-tls podName:be593a74-ef4a-4604-a64f-5e8ee0722fb7 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:16.507790524 +0000 UTC m=+35.608297062 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/be593a74-ef4a-4604-a64f-5e8ee0722fb7-metrics-tls") pod "dns-default-tmddp" (UID: "be593a74-ef4a-4604-a64f-5e8ee0722fb7") : secret "dns-default-metrics-tls" not found
Jul 28 10:08:16.008025 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:16.007962    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2vwd\" (UniqueName: \"kubernetes.io/projected/eb3e24da-20ea-46a2-8a54-3a518d710941-kube-api-access-j2vwd\") pod \"ingress-canary-zqw4b\" (UID: \"eb3e24da-20ea-46a2-8a54-3a518d710941\") " pod="openshift-ingress-canary/ingress-canary-zqw4b"
Jul 28 10:08:16.008025 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:16.008015    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be593a74-ef4a-4604-a64f-5e8ee0722fb7-tmp-dir\") pod \"dns-default-tmddp\" (UID: \"be593a74-ef4a-4604-a64f-5e8ee0722fb7\") " pod="openshift-dns/dns-default-tmddp"
Jul 28 10:08:16.008124 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:16.008060    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/be593a74-ef4a-4604-a64f-5e8ee0722fb7-config-volume\") pod \"dns-default-tmddp\" (UID: \"be593a74-ef4a-4604-a64f-5e8ee0722fb7\") " pod="openshift-dns/dns-default-tmddp"
Jul 28 10:08:16.008401 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:16.008372    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be593a74-ef4a-4604-a64f-5e8ee0722fb7-tmp-dir\") pod \"dns-default-tmddp\" (UID: \"be593a74-ef4a-4604-a64f-5e8ee0722fb7\") " pod="openshift-dns/dns-default-tmddp"
Jul 28 10:08:16.008749 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:16.008719    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/be593a74-ef4a-4604-a64f-5e8ee0722fb7-config-volume\") pod \"dns-default-tmddp\" (UID: \"be593a74-ef4a-4604-a64f-5e8ee0722fb7\") " pod="openshift-dns/dns-default-tmddp"
Jul 28 10:08:16.012407 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:16.012381    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eb3e24da-20ea-46a2-8a54-3a518d710941-cert\") pod \"ingress-canary-zqw4b\" (UID: \"eb3e24da-20ea-46a2-8a54-3a518d710941\") " pod="openshift-ingress-canary/ingress-canary-zqw4b"
Jul 28 10:08:16.016225 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:16.016202    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2vwd\" (UniqueName: \"kubernetes.io/projected/eb3e24da-20ea-46a2-8a54-3a518d710941-kube-api-access-j2vwd\") pod \"ingress-canary-zqw4b\" (UID: \"eb3e24da-20ea-46a2-8a54-3a518d710941\") " pod="openshift-ingress-canary/ingress-canary-zqw4b"
Jul 28 10:08:16.018090 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:16.018070    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56fcd\" (UniqueName: \"kubernetes.io/projected/be593a74-ef4a-4604-a64f-5e8ee0722fb7-kube-api-access-56fcd\") pod \"dns-default-tmddp\" (UID: \"be593a74-ef4a-4604-a64f-5e8ee0722fb7\") " pod="openshift-dns/dns-default-tmddp"
Jul 28 10:08:16.073282 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:16.073249    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zqw4b"
Jul 28 10:08:16.218017 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:16.217829    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zqw4b"]
Jul 28 10:08:16.432813 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:16.432708    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-64pm8"
Jul 28 10:08:16.432813 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:16.432726    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7dfzm"
Jul 28 10:08:16.432813 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:16.432735    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldq5b"
Jul 28 10:08:16.435569 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:16.435538    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 10:08:16.436149 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:16.436129    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ltg76\""
Jul 28 10:08:16.436300 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:16.436217    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rf2fq\""
Jul 28 10:08:16.436384 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:16.436364    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 10:08:16.436455 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:16.436434    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 10:08:16.436638 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:16.436622    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 10:08:16.511446 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:16.511399    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/be593a74-ef4a-4604-a64f-5e8ee0722fb7-metrics-tls\") pod \"dns-default-tmddp\" (UID: \"be593a74-ef4a-4604-a64f-5e8ee0722fb7\") " pod="openshift-dns/dns-default-tmddp"
Jul 28 10:08:16.511578 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:16.511562    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 10:08:16.511642 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:16.511626    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be593a74-ef4a-4604-a64f-5e8ee0722fb7-metrics-tls podName:be593a74-ef4a-4604-a64f-5e8ee0722fb7 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:17.511608009 +0000 UTC m=+36.612114542 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/be593a74-ef4a-4604-a64f-5e8ee0722fb7-metrics-tls") pod "dns-default-tmddp" (UID: "be593a74-ef4a-4604-a64f-5e8ee0722fb7") : secret "dns-default-metrics-tls" not found
Jul 28 10:08:17.248707 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:17.248658    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:17.351527 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:17.351492    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:17.519104 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:17.519026    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/be593a74-ef4a-4604-a64f-5e8ee0722fb7-metrics-tls\") pod \"dns-default-tmddp\" (UID: \"be593a74-ef4a-4604-a64f-5e8ee0722fb7\") " pod="openshift-dns/dns-default-tmddp"
Jul 28 10:08:17.519246 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:17.519144    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 10:08:17.519246 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:17.519225    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be593a74-ef4a-4604-a64f-5e8ee0722fb7-metrics-tls podName:be593a74-ef4a-4604-a64f-5e8ee0722fb7 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:19.5192045 +0000 UTC m=+38.619711056 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/be593a74-ef4a-4604-a64f-5e8ee0722fb7-metrics-tls") pod "dns-default-tmddp" (UID: "be593a74-ef4a-4604-a64f-5e8ee0722fb7") : secret "dns-default-metrics-tls" not found
Jul 28 10:08:17.593887 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:08:17.593858    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb3e24da_20ea_46a2_8a54_3a518d710941.slice/crio-fd7f627c17317127dabb8153a47d74aef7f327771c2878ac18572e5906383399 WatchSource:0}: Error finding container fd7f627c17317127dabb8153a47d74aef7f327771c2878ac18572e5906383399: Status 404 returned error can't find the container with id fd7f627c17317127dabb8153a47d74aef7f327771c2878ac18572e5906383399
Jul 28 10:08:18.592752 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:18.592717    2583 generic.go:356] "Generic (PLEG): container finished" podID="675b6659-b757-4eca-ba86-28b21af90501" containerID="e92e1b8aad3d57f4aacd87d5e42c11a12e7ae30eed620cb2476d98091bf50c23" exitCode=0
Jul 28 10:08:18.593174 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:18.592811    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-545mm" event={"ID":"675b6659-b757-4eca-ba86-28b21af90501","Type":"ContainerDied","Data":"e92e1b8aad3d57f4aacd87d5e42c11a12e7ae30eed620cb2476d98091bf50c23"}
Jul 28 10:08:18.594030 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:18.594006    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zqw4b" event={"ID":"eb3e24da-20ea-46a2-8a54-3a518d710941","Type":"ContainerStarted","Data":"fd7f627c17317127dabb8153a47d74aef7f327771c2878ac18572e5906383399"}
Jul 28 10:08:18.818229 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:18.818203    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-ccv4b"]
Jul 28 10:08:18.830318 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:18.830298    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-ccv4b"
Jul 28 10:08:18.833541 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:18.832676    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-vzfmh\""
Jul 28 10:08:18.833541 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:18.832676    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 28 10:08:18.833541 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:18.832861    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 28 10:08:18.833541 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:18.833087    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-ccv4b"]
Jul 28 10:08:18.930646 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:18.930616    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f45e9c0a-7761-4581-8ae8-dd652ec7846f-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-ccv4b\" (UID: \"f45e9c0a-7761-4581-8ae8-dd652ec7846f\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-ccv4b"
Jul 28 10:08:18.930808 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:18.930726    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f45e9c0a-7761-4581-8ae8-dd652ec7846f-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-ccv4b\" (UID: \"f45e9c0a-7761-4581-8ae8-dd652ec7846f\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-ccv4b"
Jul 28 10:08:18.946660 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:18.946635    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:19.010151 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:19.010120    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:19.031157 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:19.031129    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f45e9c0a-7761-4581-8ae8-dd652ec7846f-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-ccv4b\" (UID: \"f45e9c0a-7761-4581-8ae8-dd652ec7846f\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-ccv4b"
Jul 28 10:08:19.031298 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:19.031225    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f45e9c0a-7761-4581-8ae8-dd652ec7846f-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-ccv4b\" (UID: \"f45e9c0a-7761-4581-8ae8-dd652ec7846f\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-ccv4b"
Jul 28 10:08:19.031362 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:19.031342    2583 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 28 10:08:19.031418 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:19.031404    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f45e9c0a-7761-4581-8ae8-dd652ec7846f-networking-console-plugin-cert podName:f45e9c0a-7761-4581-8ae8-dd652ec7846f nodeName:}" failed. No retries permitted until 2026-07-28 10:08:19.531386981 +0000 UTC m=+38.631893516 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f45e9c0a-7761-4581-8ae8-dd652ec7846f-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-ccv4b" (UID: "f45e9c0a-7761-4581-8ae8-dd652ec7846f") : secret "networking-console-plugin-cert" not found
Jul 28 10:08:19.031776 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:19.031754    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f45e9c0a-7761-4581-8ae8-dd652ec7846f-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-ccv4b\" (UID: \"f45e9c0a-7761-4581-8ae8-dd652ec7846f\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-ccv4b"
Jul 28 10:08:19.506554 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:19.506517    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-h9wkh"]
Jul 28 10:08:19.520654 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:19.520625    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-h9wkh"
Jul 28 10:08:19.522789 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:19.522767    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 28 10:08:19.522959 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:19.522872    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 28 10:08:19.522959 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:19.522929    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-n7kvl\""
Jul 28 10:08:19.524274 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:19.524058    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 28 10:08:19.524498 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:19.524317    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 28 10:08:19.524763 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:19.524733    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-h9wkh"]
Jul 28 10:08:19.533647 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:19.533619    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f45e9c0a-7761-4581-8ae8-dd652ec7846f-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-ccv4b\" (UID: \"f45e9c0a-7761-4581-8ae8-dd652ec7846f\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-ccv4b"
Jul 28 10:08:19.533757 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:19.533741    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/be593a74-ef4a-4604-a64f-5e8ee0722fb7-metrics-tls\") pod \"dns-default-tmddp\" (UID: \"be593a74-ef4a-4604-a64f-5e8ee0722fb7\") " pod="openshift-dns/dns-default-tmddp"
Jul 28 10:08:19.533816 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:19.533772    2583 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 28 10:08:19.533866 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:19.533837    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f45e9c0a-7761-4581-8ae8-dd652ec7846f-networking-console-plugin-cert podName:f45e9c0a-7761-4581-8ae8-dd652ec7846f nodeName:}" failed. No retries permitted until 2026-07-28 10:08:20.533818206 +0000 UTC m=+39.634324744 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f45e9c0a-7761-4581-8ae8-dd652ec7846f-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-ccv4b" (UID: "f45e9c0a-7761-4581-8ae8-dd652ec7846f") : secret "networking-console-plugin-cert" not found
Jul 28 10:08:19.533866 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:19.533842    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 10:08:19.533963 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:19.533884    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be593a74-ef4a-4604-a64f-5e8ee0722fb7-metrics-tls podName:be593a74-ef4a-4604-a64f-5e8ee0722fb7 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:23.533868027 +0000 UTC m=+42.634374576 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/be593a74-ef4a-4604-a64f-5e8ee0722fb7-metrics-tls") pod "dns-default-tmddp" (UID: "be593a74-ef4a-4604-a64f-5e8ee0722fb7") : secret "dns-default-metrics-tls" not found
Jul 28 10:08:19.601035 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:19.600993    2583 generic.go:356] "Generic (PLEG): container finished" podID="675b6659-b757-4eca-ba86-28b21af90501" containerID="45ca14c5ad7ebb9db283e1f9f4e52e59d5f99ecd86f306778b1aa3a1f23a0c3c" exitCode=0
Jul 28 10:08:19.601486 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:19.601056    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-545mm" event={"ID":"675b6659-b757-4eca-ba86-28b21af90501","Type":"ContainerDied","Data":"45ca14c5ad7ebb9db283e1f9f4e52e59d5f99ecd86f306778b1aa3a1f23a0c3c"}
Jul 28 10:08:19.634649 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:19.634616    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1f82a539-9c16-4566-96ae-7d63e58d46d2-data-volume\") pod \"insights-runtime-extractor-h9wkh\" (UID: \"1f82a539-9c16-4566-96ae-7d63e58d46d2\") " pod="openshift-insights/insights-runtime-extractor-h9wkh"
Jul 28 10:08:19.634781 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:19.634658    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1f82a539-9c16-4566-96ae-7d63e58d46d2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h9wkh\" (UID: \"1f82a539-9c16-4566-96ae-7d63e58d46d2\") " pod="openshift-insights/insights-runtime-extractor-h9wkh"
Jul 28 10:08:19.634781 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:19.634678    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1f82a539-9c16-4566-96ae-7d63e58d46d2-crio-socket\") pod \"insights-runtime-extractor-h9wkh\" (UID: \"1f82a539-9c16-4566-96ae-7d63e58d46d2\") " pod="openshift-insights/insights-runtime-extractor-h9wkh"
Jul 28 10:08:19.634781 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:19.634718    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcjlb\" (UniqueName: \"kubernetes.io/projected/1f82a539-9c16-4566-96ae-7d63e58d46d2-kube-api-access-vcjlb\") pod \"insights-runtime-extractor-h9wkh\" (UID: \"1f82a539-9c16-4566-96ae-7d63e58d46d2\") " pod="openshift-insights/insights-runtime-extractor-h9wkh"
Jul 28 10:08:19.634781 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:19.634745    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1f82a539-9c16-4566-96ae-7d63e58d46d2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h9wkh\" (UID: \"1f82a539-9c16-4566-96ae-7d63e58d46d2\") " pod="openshift-insights/insights-runtime-extractor-h9wkh"
Jul 28 10:08:19.735836 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:19.735645    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1f82a539-9c16-4566-96ae-7d63e58d46d2-data-volume\") pod \"insights-runtime-extractor-h9wkh\" (UID: \"1f82a539-9c16-4566-96ae-7d63e58d46d2\") " pod="openshift-insights/insights-runtime-extractor-h9wkh"
Jul 28 10:08:19.735931 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:19.735867    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1f82a539-9c16-4566-96ae-7d63e58d46d2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h9wkh\" (UID: \"1f82a539-9c16-4566-96ae-7d63e58d46d2\") " pod="openshift-insights/insights-runtime-extractor-h9wkh"
Jul 28 10:08:19.735931 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:19.735904    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1f82a539-9c16-4566-96ae-7d63e58d46d2-crio-socket\") pod \"insights-runtime-extractor-h9wkh\" (UID: \"1f82a539-9c16-4566-96ae-7d63e58d46d2\") " pod="openshift-insights/insights-runtime-extractor-h9wkh"
Jul 28 10:08:19.736043 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:19.735951    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcjlb\" (UniqueName: \"kubernetes.io/projected/1f82a539-9c16-4566-96ae-7d63e58d46d2-kube-api-access-vcjlb\") pod \"insights-runtime-extractor-h9wkh\" (UID: \"1f82a539-9c16-4566-96ae-7d63e58d46d2\") " pod="openshift-insights/insights-runtime-extractor-h9wkh"
Jul 28 10:08:19.736043 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:19.735979    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1f82a539-9c16-4566-96ae-7d63e58d46d2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h9wkh\" (UID: \"1f82a539-9c16-4566-96ae-7d63e58d46d2\") " pod="openshift-insights/insights-runtime-extractor-h9wkh"
Jul 28 10:08:19.736043 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:19.736007    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 10:08:19.736199 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:19.736072    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f82a539-9c16-4566-96ae-7d63e58d46d2-insights-runtime-extractor-tls podName:1f82a539-9c16-4566-96ae-7d63e58d46d2 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:20.236053217 +0000 UTC m=+39.336559761 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1f82a539-9c16-4566-96ae-7d63e58d46d2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-h9wkh" (UID: "1f82a539-9c16-4566-96ae-7d63e58d46d2") : secret "insights-runtime-extractor-tls" not found
Jul 28 10:08:19.736199 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:19.736167    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1f82a539-9c16-4566-96ae-7d63e58d46d2-crio-socket\") pod \"insights-runtime-extractor-h9wkh\" (UID: \"1f82a539-9c16-4566-96ae-7d63e58d46d2\") " pod="openshift-insights/insights-runtime-extractor-h9wkh"
Jul 28 10:08:19.736572 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:19.736551    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 10:08:19.736674 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:19.736609    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1f82a539-9c16-4566-96ae-7d63e58d46d2-kube-rbac-proxy-cm podName:1f82a539-9c16-4566-96ae-7d63e58d46d2 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:20.236597177 +0000 UTC m=+39.337103708 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1f82a539-9c16-4566-96ae-7d63e58d46d2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-h9wkh" (UID: "1f82a539-9c16-4566-96ae-7d63e58d46d2") : configmap "kube-rbac-proxy" not found
Jul 28 10:08:19.744905 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:19.744873    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1f82a539-9c16-4566-96ae-7d63e58d46d2-data-volume\") pod \"insights-runtime-extractor-h9wkh\" (UID: \"1f82a539-9c16-4566-96ae-7d63e58d46d2\") " pod="openshift-insights/insights-runtime-extractor-h9wkh"
Jul 28 10:08:19.759572 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:19.759548    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcjlb\" (UniqueName: \"kubernetes.io/projected/1f82a539-9c16-4566-96ae-7d63e58d46d2-kube-api-access-vcjlb\") pod \"insights-runtime-extractor-h9wkh\" (UID: \"1f82a539-9c16-4566-96ae-7d63e58d46d2\") " pod="openshift-insights/insights-runtime-extractor-h9wkh"
Jul 28 10:08:19.763839 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:19.763816    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:19.828635 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:19.828606    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:19.909318 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:19.909297    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:20.240878 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:20.240843    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1f82a539-9c16-4566-96ae-7d63e58d46d2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h9wkh\" (UID: \"1f82a539-9c16-4566-96ae-7d63e58d46d2\") " pod="openshift-insights/insights-runtime-extractor-h9wkh"
Jul 28 10:08:20.241024 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:20.240905    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1f82a539-9c16-4566-96ae-7d63e58d46d2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h9wkh\" (UID: \"1f82a539-9c16-4566-96ae-7d63e58d46d2\") " pod="openshift-insights/insights-runtime-extractor-h9wkh"
Jul 28 10:08:20.241024 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:20.241015    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 10:08:20.241092 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:20.241054    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 10:08:20.241092 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:20.241070    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f82a539-9c16-4566-96ae-7d63e58d46d2-insights-runtime-extractor-tls podName:1f82a539-9c16-4566-96ae-7d63e58d46d2 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:21.241056957 +0000 UTC m=+40.341563492 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1f82a539-9c16-4566-96ae-7d63e58d46d2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-h9wkh" (UID: "1f82a539-9c16-4566-96ae-7d63e58d46d2") : secret "insights-runtime-extractor-tls" not found
Jul 28 10:08:20.241157 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:20.241106    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1f82a539-9c16-4566-96ae-7d63e58d46d2-kube-rbac-proxy-cm podName:1f82a539-9c16-4566-96ae-7d63e58d46d2 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:21.241093354 +0000 UTC m=+40.341599886 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1f82a539-9c16-4566-96ae-7d63e58d46d2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-h9wkh" (UID: "1f82a539-9c16-4566-96ae-7d63e58d46d2") : configmap "kube-rbac-proxy" not found
Jul 28 10:08:20.328400 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:20.328339    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:20.542687 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:20.542657    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f45e9c0a-7761-4581-8ae8-dd652ec7846f-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-ccv4b\" (UID: \"f45e9c0a-7761-4581-8ae8-dd652ec7846f\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-ccv4b"
Jul 28 10:08:20.542839 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:20.542770    2583 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 28 10:08:20.542839 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:20.542822    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f45e9c0a-7761-4581-8ae8-dd652ec7846f-networking-console-plugin-cert podName:f45e9c0a-7761-4581-8ae8-dd652ec7846f nodeName:}" failed. No retries permitted until 2026-07-28 10:08:22.542809734 +0000 UTC m=+41.643316266 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f45e9c0a-7761-4581-8ae8-dd652ec7846f-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-ccv4b" (UID: "f45e9c0a-7761-4581-8ae8-dd652ec7846f") : secret "networking-console-plugin-cert" not found
Jul 28 10:08:20.604952 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:20.604876    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zqw4b" event={"ID":"eb3e24da-20ea-46a2-8a54-3a518d710941","Type":"ContainerStarted","Data":"6acaacd90df0ec177b4a0d37582d9bed343d7e1c70a4055d62719f842a737e8e"}
Jul 28 10:08:20.607907 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:20.607882    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-545mm" event={"ID":"675b6659-b757-4eca-ba86-28b21af90501","Type":"ContainerStarted","Data":"42a4f62bdf2187364e786314854079c538002c5213169c0d250978a578190669"}
Jul 28 10:08:20.632523 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:20.632481    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-zqw4b" podStartSLOduration=3.381370009 podStartE2EDuration="5.632466431s" totalImagesPullingTime="2.251096422s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:08:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:20.631890546 +0000 UTC m=+39.732397123" watchObservedRunningTime="2026-07-28 10:08:20.632466431 +0000 UTC m=+39.732972963"
Jul 28 10:08:20.680577 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:20.680523    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-545mm" podStartSLOduration=9.829605117 podStartE2EDuration="39.68050133s" totalImagesPullingTime="29.850896213s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:07:41 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:20.678957927 +0000 UTC m=+39.779464501" watchObservedRunningTime="2026-07-28 10:08:20.68050133 +0000 UTC m=+39.781007885"
Jul 28 10:08:20.746826 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:20.746791    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:20.834593 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:20.834570    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:20.923089 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:20.923026    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:20.924885 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:20.924866    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6b5bff677c-7d2rn"]
Jul 28 10:08:20.958697 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:20.958627    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6b5bff677c-7d2rn"]
Jul 28 10:08:20.958697 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:20.958696    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b5bff677c-7d2rn"
Jul 28 10:08:20.962276 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:20.962260    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 28 10:08:20.962757 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:20.962740    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 28 10:08:20.963058 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:20.963044    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 28 10:08:20.963309 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:20.963295    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-8x85w\""
Jul 28 10:08:20.968997 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:20.968979    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 28 10:08:21.046441 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:21.046414    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec-registry-certificates\") pod \"image-registry-6b5bff677c-7d2rn\" (UID: \"b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec\") " pod="openshift-image-registry/image-registry-6b5bff677c-7d2rn"
Jul 28 10:08:21.046552 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:21.046444    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec-installation-pull-secrets\") pod \"image-registry-6b5bff677c-7d2rn\" (UID: \"b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec\") " pod="openshift-image-registry/image-registry-6b5bff677c-7d2rn"
Jul 28 10:08:21.046552 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:21.046467    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec-bound-sa-token\") pod \"image-registry-6b5bff677c-7d2rn\" (UID: \"b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec\") " pod="openshift-image-registry/image-registry-6b5bff677c-7d2rn"
Jul 28 10:08:21.046552 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:21.046497    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65787\" (UniqueName: \"kubernetes.io/projected/b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec-kube-api-access-65787\") pod \"image-registry-6b5bff677c-7d2rn\" (UID: \"b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec\") " pod="openshift-image-registry/image-registry-6b5bff677c-7d2rn"
Jul 28 10:08:21.046733 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:21.046597    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec-trusted-ca\") pod \"image-registry-6b5bff677c-7d2rn\" (UID: \"b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec\") " pod="openshift-image-registry/image-registry-6b5bff677c-7d2rn"
Jul 28 10:08:21.046733 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:21.046638    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec-registry-tls\") pod \"image-registry-6b5bff677c-7d2rn\" (UID: \"b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec\") " pod="openshift-image-registry/image-registry-6b5bff677c-7d2rn"
Jul 28 10:08:21.046733 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:21.046716    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec-ca-trust-extracted\") pod \"image-registry-6b5bff677c-7d2rn\" (UID: \"b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec\") " pod="openshift-image-registry/image-registry-6b5bff677c-7d2rn"
Jul 28 10:08:21.046856 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:21.046759    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec-image-registry-private-configuration\") pod \"image-registry-6b5bff677c-7d2rn\" (UID: \"b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec\") " pod="openshift-image-registry/image-registry-6b5bff677c-7d2rn"
Jul 28 10:08:21.147560 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:21.147529    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65787\" (UniqueName: \"kubernetes.io/projected/b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec-kube-api-access-65787\") pod \"image-registry-6b5bff677c-7d2rn\" (UID: \"b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec\") " pod="openshift-image-registry/image-registry-6b5bff677c-7d2rn"
Jul 28 10:08:21.147678 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:21.147570    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec-trusted-ca\") pod \"image-registry-6b5bff677c-7d2rn\" (UID: \"b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec\") " pod="openshift-image-registry/image-registry-6b5bff677c-7d2rn"
Jul 28 10:08:21.147678 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:21.147593    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec-registry-tls\") pod \"image-registry-6b5bff677c-7d2rn\" (UID: \"b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec\") " pod="openshift-image-registry/image-registry-6b5bff677c-7d2rn"
Jul 28 10:08:21.147678 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:21.147672    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 10:08:21.147809 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:21.147683    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b5bff677c-7d2rn: secret "image-registry-tls" not found
Jul 28 10:08:21.147809 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:21.147730    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec-registry-tls podName:b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec nodeName:}" failed. No retries permitted until 2026-07-28 10:08:21.647712774 +0000 UTC m=+40.748219320 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec-registry-tls") pod "image-registry-6b5bff677c-7d2rn" (UID: "b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec") : secret "image-registry-tls" not found
Jul 28 10:08:21.147809 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:21.147761    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec-ca-trust-extracted\") pod \"image-registry-6b5bff677c-7d2rn\" (UID: \"b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec\") " pod="openshift-image-registry/image-registry-6b5bff677c-7d2rn"
Jul 28 10:08:21.147809 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:21.147787    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec-image-registry-private-configuration\") pod \"image-registry-6b5bff677c-7d2rn\" (UID: \"b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec\") " pod="openshift-image-registry/image-registry-6b5bff677c-7d2rn"
Jul 28 10:08:21.148004 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:21.147855    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec-registry-certificates\") pod \"image-registry-6b5bff677c-7d2rn\" (UID: \"b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec\") " pod="openshift-image-registry/image-registry-6b5bff677c-7d2rn"
Jul 28 10:08:21.148004 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:21.147885    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec-installation-pull-secrets\") pod \"image-registry-6b5bff677c-7d2rn\" (UID: \"b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec\") " pod="openshift-image-registry/image-registry-6b5bff677c-7d2rn"
Jul 28 10:08:21.148004 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:21.147913    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec-bound-sa-token\") pod \"image-registry-6b5bff677c-7d2rn\" (UID: \"b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec\") " pod="openshift-image-registry/image-registry-6b5bff677c-7d2rn"
Jul 28 10:08:21.148146 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:21.148106    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec-ca-trust-extracted\") pod \"image-registry-6b5bff677c-7d2rn\" (UID: \"b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec\") " pod="openshift-image-registry/image-registry-6b5bff677c-7d2rn"
Jul 28 10:08:21.148436 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:21.148407    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec-registry-certificates\") pod \"image-registry-6b5bff677c-7d2rn\" (UID: \"b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec\") " pod="openshift-image-registry/image-registry-6b5bff677c-7d2rn"
Jul 28 10:08:21.148968 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:21.148948    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec-trusted-ca\") pod \"image-registry-6b5bff677c-7d2rn\" (UID: \"b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec\") " pod="openshift-image-registry/image-registry-6b5bff677c-7d2rn"
Jul 28 10:08:21.151032 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:21.151012    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec-installation-pull-secrets\") pod \"image-registry-6b5bff677c-7d2rn\" (UID: \"b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec\") " pod="openshift-image-registry/image-registry-6b5bff677c-7d2rn"
Jul 28 10:08:21.151120 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:21.151101    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec-image-registry-private-configuration\") pod \"image-registry-6b5bff677c-7d2rn\" (UID: \"b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec\") " pod="openshift-image-registry/image-registry-6b5bff677c-7d2rn"
Jul 28 10:08:21.156570 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:21.156547    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65787\" (UniqueName: \"kubernetes.io/projected/b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec-kube-api-access-65787\") pod \"image-registry-6b5bff677c-7d2rn\" (UID: \"b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec\") " pod="openshift-image-registry/image-registry-6b5bff677c-7d2rn"
Jul 28 10:08:21.156855 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:21.156838    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec-bound-sa-token\") pod \"image-registry-6b5bff677c-7d2rn\" (UID: \"b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec\") " pod="openshift-image-registry/image-registry-6b5bff677c-7d2rn"
Jul 28 10:08:21.248447 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:21.248381    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1f82a539-9c16-4566-96ae-7d63e58d46d2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h9wkh\" (UID: \"1f82a539-9c16-4566-96ae-7d63e58d46d2\") " pod="openshift-insights/insights-runtime-extractor-h9wkh"
Jul 28 10:08:21.248567 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:21.248492    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 10:08:21.248567 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:21.248498    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1f82a539-9c16-4566-96ae-7d63e58d46d2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h9wkh\" (UID: \"1f82a539-9c16-4566-96ae-7d63e58d46d2\") " pod="openshift-insights/insights-runtime-extractor-h9wkh"
Jul 28 10:08:21.248567 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:21.248548    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1f82a539-9c16-4566-96ae-7d63e58d46d2-kube-rbac-proxy-cm podName:1f82a539-9c16-4566-96ae-7d63e58d46d2 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:23.248532902 +0000 UTC m=+42.349039436 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1f82a539-9c16-4566-96ae-7d63e58d46d2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-h9wkh" (UID: "1f82a539-9c16-4566-96ae-7d63e58d46d2") : configmap "kube-rbac-proxy" not found
Jul 28 10:08:21.248663 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:21.248594    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 10:08:21.248663 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:21.248635    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f82a539-9c16-4566-96ae-7d63e58d46d2-insights-runtime-extractor-tls podName:1f82a539-9c16-4566-96ae-7d63e58d46d2 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:23.248625021 +0000 UTC m=+42.349131554 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1f82a539-9c16-4566-96ae-7d63e58d46d2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-h9wkh" (UID: "1f82a539-9c16-4566-96ae-7d63e58d46d2") : secret "insights-runtime-extractor-tls" not found
Jul 28 10:08:21.298108 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:21.298085    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nxpwb_42229f5e-156f-4abd-a2ee-700c9f340775/dns-node-resolver/0.log"
Jul 28 10:08:21.314127 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:21.314101    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:21.397882 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:21.397856    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:21.650771 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:21.650694    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec-registry-tls\") pod \"image-registry-6b5bff677c-7d2rn\" (UID: \"b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec\") " pod="openshift-image-registry/image-registry-6b5bff677c-7d2rn"
Jul 28 10:08:21.651210 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:21.650843    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 10:08:21.651210 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:21.650857    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b5bff677c-7d2rn: secret "image-registry-tls" not found
Jul 28 10:08:21.651210 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:21.650907    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec-registry-tls podName:b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec nodeName:}" failed. No retries permitted until 2026-07-28 10:08:22.650892838 +0000 UTC m=+41.751399373 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec-registry-tls") pod "image-registry-6b5bff677c-7d2rn" (UID: "b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec") : secret "image-registry-tls" not found
Jul 28 10:08:22.094872 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:22.094797    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-m8tpf"]
Jul 28 10:08:22.126454 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:22.126411    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:22.129624 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:22.129609    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-m8tpf"]
Jul 28 10:08:22.129693 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:22.129683    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-m8tpf"
Jul 28 10:08:22.134327 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:22.134311    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 28 10:08:22.134616 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:22.134598    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 28 10:08:22.135130 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:22.135110    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-hvm5s\""
Jul 28 10:08:22.135226 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:22.135172    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 28 10:08:22.135226 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:22.135194    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 28 10:08:22.253660 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:22.253637    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vhmk\" (UniqueName: \"kubernetes.io/projected/7289ab6a-07ff-4549-ac66-5a69f27b5163-kube-api-access-2vhmk\") pod \"service-ca-658d7b7c78-m8tpf\" (UID: \"7289ab6a-07ff-4549-ac66-5a69f27b5163\") " pod="openshift-service-ca/service-ca-658d7b7c78-m8tpf"
Jul 28 10:08:22.253753 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:22.253689    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7289ab6a-07ff-4549-ac66-5a69f27b5163-signing-cabundle\") pod \"service-ca-658d7b7c78-m8tpf\" (UID: \"7289ab6a-07ff-4549-ac66-5a69f27b5163\") " pod="openshift-service-ca/service-ca-658d7b7c78-m8tpf"
Jul 28 10:08:22.253753 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:22.253742    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7289ab6a-07ff-4549-ac66-5a69f27b5163-signing-key\") pod \"service-ca-658d7b7c78-m8tpf\" (UID: \"7289ab6a-07ff-4549-ac66-5a69f27b5163\") " pod="openshift-service-ca/service-ca-658d7b7c78-m8tpf"
Jul 28 10:08:22.298377 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:22.298355    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-v94ww_eeccf40b-0096-40d3-b231-d7ce00688cc9/node-ca/0.log"
Jul 28 10:08:22.354746 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:22.354698    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vhmk\" (UniqueName: \"kubernetes.io/projected/7289ab6a-07ff-4549-ac66-5a69f27b5163-kube-api-access-2vhmk\") pod \"service-ca-658d7b7c78-m8tpf\" (UID: \"7289ab6a-07ff-4549-ac66-5a69f27b5163\") " pod="openshift-service-ca/service-ca-658d7b7c78-m8tpf"
Jul 28 10:08:22.354871 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:22.354757    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7289ab6a-07ff-4549-ac66-5a69f27b5163-signing-cabundle\") pod \"service-ca-658d7b7c78-m8tpf\" (UID: \"7289ab6a-07ff-4549-ac66-5a69f27b5163\") " pod="openshift-service-ca/service-ca-658d7b7c78-m8tpf"
Jul 28 10:08:22.354871 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:22.354787    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7289ab6a-07ff-4549-ac66-5a69f27b5163-signing-key\") pod \"service-ca-658d7b7c78-m8tpf\" (UID: \"7289ab6a-07ff-4549-ac66-5a69f27b5163\") " pod="openshift-service-ca/service-ca-658d7b7c78-m8tpf"
Jul 28 10:08:22.355385 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:22.355367    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7289ab6a-07ff-4549-ac66-5a69f27b5163-signing-cabundle\") pod \"service-ca-658d7b7c78-m8tpf\" (UID: \"7289ab6a-07ff-4549-ac66-5a69f27b5163\") " pod="openshift-service-ca/service-ca-658d7b7c78-m8tpf"
Jul 28 10:08:22.357038 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:22.357019    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7289ab6a-07ff-4549-ac66-5a69f27b5163-signing-key\") pod \"service-ca-658d7b7c78-m8tpf\" (UID: \"7289ab6a-07ff-4549-ac66-5a69f27b5163\") " pod="openshift-service-ca/service-ca-658d7b7c78-m8tpf"
Jul 28 10:08:22.363469 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:22.363450    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vhmk\" (UniqueName: \"kubernetes.io/projected/7289ab6a-07ff-4549-ac66-5a69f27b5163-kube-api-access-2vhmk\") pod \"service-ca-658d7b7c78-m8tpf\" (UID: \"7289ab6a-07ff-4549-ac66-5a69f27b5163\") " pod="openshift-service-ca/service-ca-658d7b7c78-m8tpf"
Jul 28 10:08:22.438683 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:22.438663    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-m8tpf"
Jul 28 10:08:22.559920 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:22.558671    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f45e9c0a-7761-4581-8ae8-dd652ec7846f-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-ccv4b\" (UID: \"f45e9c0a-7761-4581-8ae8-dd652ec7846f\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-ccv4b"
Jul 28 10:08:22.559920 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:22.558849    2583 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 28 10:08:22.559920 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:22.558926    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f45e9c0a-7761-4581-8ae8-dd652ec7846f-networking-console-plugin-cert podName:f45e9c0a-7761-4581-8ae8-dd652ec7846f nodeName:}" failed. No retries permitted until 2026-07-28 10:08:26.558905435 +0000 UTC m=+45.659411984 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f45e9c0a-7761-4581-8ae8-dd652ec7846f-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-ccv4b" (UID: "f45e9c0a-7761-4581-8ae8-dd652ec7846f") : secret "networking-console-plugin-cert" not found
Jul 28 10:08:22.659837 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:22.659771    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec-registry-tls\") pod \"image-registry-6b5bff677c-7d2rn\" (UID: \"b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec\") " pod="openshift-image-registry/image-registry-6b5bff677c-7d2rn"
Jul 28 10:08:22.660134 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:22.659916    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 10:08:22.660134 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:22.659930    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b5bff677c-7d2rn: secret "image-registry-tls" not found
Jul 28 10:08:22.660134 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:22.659978    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec-registry-tls podName:b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec nodeName:}" failed. No retries permitted until 2026-07-28 10:08:24.659963581 +0000 UTC m=+43.760470113 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec-registry-tls") pod "image-registry-6b5bff677c-7d2rn" (UID: "b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec") : secret "image-registry-tls" not found
Jul 28 10:08:22.667477 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:22.667460    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-m8tpf"]
Jul 28 10:08:22.669528 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:08:22.669496    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7289ab6a_07ff_4549_ac66_5a69f27b5163.slice/crio-6995e02ada3bd7cfc7f7580e4813dc4314485371d93953039e342da1fdaaf0ca WatchSource:0}: Error finding container 6995e02ada3bd7cfc7f7580e4813dc4314485371d93953039e342da1fdaaf0ca: Status 404 returned error can't find the container with id 6995e02ada3bd7cfc7f7580e4813dc4314485371d93953039e342da1fdaaf0ca
Jul 28 10:08:23.263937 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:23.263909    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1f82a539-9c16-4566-96ae-7d63e58d46d2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h9wkh\" (UID: \"1f82a539-9c16-4566-96ae-7d63e58d46d2\") " pod="openshift-insights/insights-runtime-extractor-h9wkh"
Jul 28 10:08:23.264082 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:23.263956    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1f82a539-9c16-4566-96ae-7d63e58d46d2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h9wkh\" (UID: \"1f82a539-9c16-4566-96ae-7d63e58d46d2\") " pod="openshift-insights/insights-runtime-extractor-h9wkh"
Jul 28 10:08:23.264082 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:23.264056    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 10:08:23.264167 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:23.264114    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f82a539-9c16-4566-96ae-7d63e58d46d2-insights-runtime-extractor-tls podName:1f82a539-9c16-4566-96ae-7d63e58d46d2 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:27.264099332 +0000 UTC m=+46.364605865 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1f82a539-9c16-4566-96ae-7d63e58d46d2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-h9wkh" (UID: "1f82a539-9c16-4566-96ae-7d63e58d46d2") : secret "insights-runtime-extractor-tls" not found
Jul 28 10:08:23.264167 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:23.264060    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 10:08:23.264263 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:23.264201    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1f82a539-9c16-4566-96ae-7d63e58d46d2-kube-rbac-proxy-cm podName:1f82a539-9c16-4566-96ae-7d63e58d46d2 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:27.26417285 +0000 UTC m=+46.364679382 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1f82a539-9c16-4566-96ae-7d63e58d46d2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-h9wkh" (UID: "1f82a539-9c16-4566-96ae-7d63e58d46d2") : configmap "kube-rbac-proxy" not found
Jul 28 10:08:23.300888 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:23.300862    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-zqw4b_eb3e24da-20ea-46a2-8a54-3a518d710941/serve-healthcheck-canary/0.log"
Jul 28 10:08:23.566707 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:23.566622    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/be593a74-ef4a-4604-a64f-5e8ee0722fb7-metrics-tls\") pod \"dns-default-tmddp\" (UID: \"be593a74-ef4a-4604-a64f-5e8ee0722fb7\") " pod="openshift-dns/dns-default-tmddp"
Jul 28 10:08:23.566836 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:23.566798    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 10:08:23.566878 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:23.566861    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be593a74-ef4a-4604-a64f-5e8ee0722fb7-metrics-tls podName:be593a74-ef4a-4604-a64f-5e8ee0722fb7 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:31.566845097 +0000 UTC m=+50.667351629 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/be593a74-ef4a-4604-a64f-5e8ee0722fb7-metrics-tls") pod "dns-default-tmddp" (UID: "be593a74-ef4a-4604-a64f-5e8ee0722fb7") : secret "dns-default-metrics-tls" not found
Jul 28 10:08:23.617280 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:23.617242    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-m8tpf" event={"ID":"7289ab6a-07ff-4549-ac66-5a69f27b5163","Type":"ContainerStarted","Data":"6995e02ada3bd7cfc7f7580e4813dc4314485371d93953039e342da1fdaaf0ca"}
Jul 28 10:08:23.824960 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:23.824813    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:24.675964 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:24.675924    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec-registry-tls\") pod \"image-registry-6b5bff677c-7d2rn\" (UID: \"b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec\") " pod="openshift-image-registry/image-registry-6b5bff677c-7d2rn"
Jul 28 10:08:24.676154 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:24.676085    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 10:08:24.676154 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:24.676103    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b5bff677c-7d2rn: secret "image-registry-tls" not found
Jul 28 10:08:24.676278 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:24.676167    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec-registry-tls podName:b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec nodeName:}" failed. No retries permitted until 2026-07-28 10:08:28.676147357 +0000 UTC m=+47.776653890 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec-registry-tls") pod "image-registry-6b5bff677c-7d2rn" (UID: "b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec") : secret "image-registry-tls" not found
Jul 28 10:08:25.623246 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:25.623209    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-m8tpf" event={"ID":"7289ab6a-07ff-4549-ac66-5a69f27b5163","Type":"ContainerStarted","Data":"8496106a700309f4f6a552f59be35a1343ce41333bab631cb44a7db178220a49"}
Jul 28 10:08:25.654061 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:25.654017    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-m8tpf" podStartSLOduration=1.330246674 podStartE2EDuration="3.654001869s" totalImagesPullingTime="2.323755195s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:08:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:25.653672085 +0000 UTC m=+44.754178639" watchObservedRunningTime="2026-07-28 10:08:25.654001869 +0000 UTC m=+44.754508422"
Jul 28 10:08:26.183078 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:26.183031    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:26.590734 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:26.590654    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f45e9c0a-7761-4581-8ae8-dd652ec7846f-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-ccv4b\" (UID: \"f45e9c0a-7761-4581-8ae8-dd652ec7846f\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-ccv4b"
Jul 28 10:08:26.590895 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:26.590755    2583 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 28 10:08:26.590895 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:26.590806    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f45e9c0a-7761-4581-8ae8-dd652ec7846f-networking-console-plugin-cert podName:f45e9c0a-7761-4581-8ae8-dd652ec7846f nodeName:}" failed. No retries permitted until 2026-07-28 10:08:34.590792909 +0000 UTC m=+53.691299442 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f45e9c0a-7761-4581-8ae8-dd652ec7846f-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-ccv4b" (UID: "f45e9c0a-7761-4581-8ae8-dd652ec7846f") : secret "networking-console-plugin-cert" not found
Jul 28 10:08:27.296759 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:27.296721    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1f82a539-9c16-4566-96ae-7d63e58d46d2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h9wkh\" (UID: \"1f82a539-9c16-4566-96ae-7d63e58d46d2\") " pod="openshift-insights/insights-runtime-extractor-h9wkh"
Jul 28 10:08:27.297240 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:27.296774    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1f82a539-9c16-4566-96ae-7d63e58d46d2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h9wkh\" (UID: \"1f82a539-9c16-4566-96ae-7d63e58d46d2\") " pod="openshift-insights/insights-runtime-extractor-h9wkh"
Jul 28 10:08:27.297240 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:27.296865    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 10:08:27.297240 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:27.296928    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f82a539-9c16-4566-96ae-7d63e58d46d2-insights-runtime-extractor-tls podName:1f82a539-9c16-4566-96ae-7d63e58d46d2 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:35.296913245 +0000 UTC m=+54.397419780 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1f82a539-9c16-4566-96ae-7d63e58d46d2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-h9wkh" (UID: "1f82a539-9c16-4566-96ae-7d63e58d46d2") : secret "insights-runtime-extractor-tls" not found
Jul 28 10:08:27.297240 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:27.297201    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1f82a539-9c16-4566-96ae-7d63e58d46d2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h9wkh\" (UID: \"1f82a539-9c16-4566-96ae-7d63e58d46d2\") " pod="openshift-insights/insights-runtime-extractor-h9wkh"
Jul 28 10:08:27.668634 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:27.668595    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:27.764218 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:27.764169    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:27.835442 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:27.835423    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:27.951666 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:27.951588    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:28.056310 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:28.056280    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:28.163160 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:28.163130    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:28.516795 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:28.516764    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:28.710642 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:28.710612    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec-registry-tls\") pod \"image-registry-6b5bff677c-7d2rn\" (UID: \"b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec\") " pod="openshift-image-registry/image-registry-6b5bff677c-7d2rn"
Jul 28 10:08:28.710788 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:28.710708    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 10:08:28.710788 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:28.710730    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b5bff677c-7d2rn: secret "image-registry-tls" not found
Jul 28 10:08:28.710788 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:28.710781    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec-registry-tls podName:b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec nodeName:}" failed. No retries permitted until 2026-07-28 10:08:36.710766244 +0000 UTC m=+55.811272777 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec-registry-tls") pod "image-registry-6b5bff677c-7d2rn" (UID: "b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec") : secret "image-registry-tls" not found
Jul 28 10:08:28.943080 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:28.943052    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:29.612964 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:29.612905    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:30.240084 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:30.240055    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:30.334166 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:30.334141    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:30.401008 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:30.400984    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:30.557147 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:30.557094    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:31.145523 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:31.145498    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:31.351152 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:31.351122    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:31.541639 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:31.541568    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:31.633304 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:31.633268    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/be593a74-ef4a-4604-a64f-5e8ee0722fb7-metrics-tls\") pod \"dns-default-tmddp\" (UID: \"be593a74-ef4a-4604-a64f-5e8ee0722fb7\") " pod="openshift-dns/dns-default-tmddp"
Jul 28 10:08:31.635453 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:31.635431    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/be593a74-ef4a-4604-a64f-5e8ee0722fb7-metrics-tls\") pod \"dns-default-tmddp\" (UID: \"be593a74-ef4a-4604-a64f-5e8ee0722fb7\") " pod="openshift-dns/dns-default-tmddp"
Jul 28 10:08:31.666015 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:31.665994    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tmddp"
Jul 28 10:08:31.747767 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:31.747736    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:31.795806 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:31.795751    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tmddp"]
Jul 28 10:08:31.801585 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:08:31.801545    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe593a74_ef4a_4604_a64f_5e8ee0722fb7.slice/crio-b57490fec3071bf6f4a8da588b39458fb2a20a2a6c6b5669776b1b052f6f37d8 WatchSource:0}: Error finding container b57490fec3071bf6f4a8da588b39458fb2a20a2a6c6b5669776b1b052f6f37d8: Status 404 returned error can't find the container with id b57490fec3071bf6f4a8da588b39458fb2a20a2a6c6b5669776b1b052f6f37d8
Jul 28 10:08:32.153483 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:32.153459    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:32.267740 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:32.267715    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:32.363728 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:32.363699    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:32.642507 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:32.642484    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tmddp" event={"ID":"be593a74-ef4a-4604-a64f-5e8ee0722fb7","Type":"ContainerStarted","Data":"b57490fec3071bf6f4a8da588b39458fb2a20a2a6c6b5669776b1b052f6f37d8"}
Jul 28 10:08:32.752599 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:32.752546    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:32.829641 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:32.829616    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:33.153703 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:33.153649    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:33.589035 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:33.588963    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:33.946702 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:33.946681    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:34.159995 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:34.159971    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:34.555764 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:34.555702    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:34.648609 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:34.648584    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tmddp" event={"ID":"be593a74-ef4a-4604-a64f-5e8ee0722fb7","Type":"ContainerStarted","Data":"b0de162c8d29a6c4001e9b7f61436f24ab5228a812cb454e30cd1c2b03229f12"}
Jul 28 10:08:34.648725 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:34.648631    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tmddp" event={"ID":"be593a74-ef4a-4604-a64f-5e8ee0722fb7","Type":"ContainerStarted","Data":"af2992c61efc641a62b56c2080044a0fd3aa301b024278242172100f66b2a566"}
Jul 28 10:08:34.648771 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:34.648729    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-tmddp"
Jul 28 10:08:34.655366 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:34.655347    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f45e9c0a-7761-4581-8ae8-dd652ec7846f-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-ccv4b\" (UID: \"f45e9c0a-7761-4581-8ae8-dd652ec7846f\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-ccv4b"
Jul 28 10:08:34.655457 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:34.655444    2583 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 28 10:08:34.655508 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:34.655499    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f45e9c0a-7761-4581-8ae8-dd652ec7846f-networking-console-plugin-cert podName:f45e9c0a-7761-4581-8ae8-dd652ec7846f nodeName:}" failed. No retries permitted until 2026-07-28 10:08:50.655486778 +0000 UTC m=+69.755993309 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f45e9c0a-7761-4581-8ae8-dd652ec7846f-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-ccv4b" (UID: "f45e9c0a-7761-4581-8ae8-dd652ec7846f") : secret "networking-console-plugin-cert" not found
Jul 28 10:08:34.668908 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:34.668865    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-tmddp" podStartSLOduration=17.53124958 podStartE2EDuration="19.668849938s" totalImagesPullingTime="2.137600358s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:08:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:34.668262664 +0000 UTC m=+53.768769219" watchObservedRunningTime="2026-07-28 10:08:34.668849938 +0000 UTC m=+53.769356493"
Jul 28 10:08:35.183266 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:35.183230    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:35.255267 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:35.255244    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:35.361545 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:35.361520    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1f82a539-9c16-4566-96ae-7d63e58d46d2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h9wkh\" (UID: \"1f82a539-9c16-4566-96ae-7d63e58d46d2\") " pod="openshift-insights/insights-runtime-extractor-h9wkh"
Jul 28 10:08:35.363754 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:35.363730    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1f82a539-9c16-4566-96ae-7d63e58d46d2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h9wkh\" (UID: \"1f82a539-9c16-4566-96ae-7d63e58d46d2\") " pod="openshift-insights/insights-runtime-extractor-h9wkh"
Jul 28 10:08:35.431722 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:35.431699    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-h9wkh"
Jul 28 10:08:35.557632 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:35.557552    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-h9wkh"]
Jul 28 10:08:35.561940 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:08:35.561914    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f82a539_9c16_4566_96ae_7d63e58d46d2.slice/crio-01ff3bb115271116dce8e584539a2f77f26538eb6a32514973b86f96b9a3ad33 WatchSource:0}: Error finding container 01ff3bb115271116dce8e584539a2f77f26538eb6a32514973b86f96b9a3ad33: Status 404 returned error can't find the container with id 01ff3bb115271116dce8e584539a2f77f26538eb6a32514973b86f96b9a3ad33
Jul 28 10:08:35.652037 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:35.652006    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-h9wkh" event={"ID":"1f82a539-9c16-4566-96ae-7d63e58d46d2","Type":"ContainerStarted","Data":"01ff3bb115271116dce8e584539a2f77f26538eb6a32514973b86f96b9a3ad33"}
Jul 28 10:08:35.758036 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:35.757972    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:35.967516 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:35.967487    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:36.358405 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:36.358284    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:36.550659 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:36.550583    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:36.770250 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:36.770219    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec-registry-tls\") pod \"image-registry-6b5bff677c-7d2rn\" (UID: \"b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec\") " pod="openshift-image-registry/image-registry-6b5bff677c-7d2rn"
Jul 28 10:08:36.773034 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:36.773014    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec-registry-tls\") pod \"image-registry-6b5bff677c-7d2rn\" (UID: \"b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec\") " pod="openshift-image-registry/image-registry-6b5bff677c-7d2rn"
Jul 28 10:08:36.868339 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:36.868297    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b5bff677c-7d2rn"
Jul 28 10:08:37.009260 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:37.009202    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6b5bff677c-7d2rn"]
Jul 28 10:08:37.015382 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:08:37.015340    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4cd4db4_c476_4634_a7ab_ea83fb2ca5ec.slice/crio-32ec514a1f0845f112f887ed20795c633c98fd309d7025af235c7e08ef691843 WatchSource:0}: Error finding container 32ec514a1f0845f112f887ed20795c633c98fd309d7025af235c7e08ef691843: Status 404 returned error can't find the container with id 32ec514a1f0845f112f887ed20795c633c98fd309d7025af235c7e08ef691843
Jul 28 10:08:37.160410 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:37.160378    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:37.372068 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:37.371985    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:37.660275 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:37.660236    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b5bff677c-7d2rn" event={"ID":"b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec","Type":"ContainerStarted","Data":"f61bb5ebc20ac69cff774dd2ce13825408fc70e16bdbd659365f424b7fbbbef9"}
Jul 28 10:08:37.660439 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:37.660290    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b5bff677c-7d2rn" event={"ID":"b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec","Type":"ContainerStarted","Data":"32ec514a1f0845f112f887ed20795c633c98fd309d7025af235c7e08ef691843"}
Jul 28 10:08:37.660439 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:37.660327    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6b5bff677c-7d2rn"
Jul 28 10:08:37.684137 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:37.684089    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6b5bff677c-7d2rn" podStartSLOduration=17.684069886 podStartE2EDuration="17.684069886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:08:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:37.682888548 +0000 UTC m=+56.783395102" watchObservedRunningTime="2026-07-28 10:08:37.684069886 +0000 UTC m=+56.784576442"
Jul 28 10:08:37.858389 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:37.858356    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:38.161359 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:38.161320    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:38.354398 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:38.354362    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:41.575424 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:41.575354    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rrxnl"
Jul 28 10:08:41.676223 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:41.676193    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-h9wkh" event={"ID":"1f82a539-9c16-4566-96ae-7d63e58d46d2","Type":"ContainerStarted","Data":"506b7f66e8f327335538ecc1a330921a98e018206b50aad79155fe544d41c767"}
Jul 28 10:08:44.654333 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:44.654301    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-tmddp"
Jul 28 10:08:45.694384 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:45.694315    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-h9wkh" event={"ID":"1f82a539-9c16-4566-96ae-7d63e58d46d2","Type":"ContainerStarted","Data":"e304ec6075af6873135fca0f4bdd354d4b93cbe0f494e5b46d2e8541f94e4cb3"}
Jul 28 10:08:46.289342 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:46.289305    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7ddccc4fc8-hfwhh"]
Jul 28 10:08:46.291459 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:46.291437    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7ddccc4fc8-hfwhh"
Jul 28 10:08:46.300073 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:46.300050    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-5kd9r\""
Jul 28 10:08:46.300719 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:46.300699    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 28 10:08:46.300804 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:46.300759    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 28 10:08:46.301041 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:46.301023    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 28 10:08:46.301116 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:46.301048    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 28 10:08:46.308962 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:46.308938    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7ddccc4fc8-hfwhh"]
Jul 28 10:08:46.353963 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:46.353939    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2de8c047-b0f7-4615-a1ab-5923a2d8141b-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7ddccc4fc8-hfwhh\" (UID: \"2de8c047-b0f7-4615-a1ab-5923a2d8141b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7ddccc4fc8-hfwhh"
Jul 28 10:08:46.354075 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:46.353985    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpt9f\" (UniqueName: \"kubernetes.io/projected/2de8c047-b0f7-4615-a1ab-5923a2d8141b-kube-api-access-gpt9f\") pod \"managed-serviceaccount-addon-agent-7ddccc4fc8-hfwhh\" (UID: \"2de8c047-b0f7-4615-a1ab-5923a2d8141b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7ddccc4fc8-hfwhh"
Jul 28 10:08:46.455098 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:46.455061    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2de8c047-b0f7-4615-a1ab-5923a2d8141b-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7ddccc4fc8-hfwhh\" (UID: \"2de8c047-b0f7-4615-a1ab-5923a2d8141b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7ddccc4fc8-hfwhh"
Jul 28 10:08:46.455245 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:46.455216    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpt9f\" (UniqueName: \"kubernetes.io/projected/2de8c047-b0f7-4615-a1ab-5923a2d8141b-kube-api-access-gpt9f\") pod \"managed-serviceaccount-addon-agent-7ddccc4fc8-hfwhh\" (UID: \"2de8c047-b0f7-4615-a1ab-5923a2d8141b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7ddccc4fc8-hfwhh"
Jul 28 10:08:46.457841 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:46.457817    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2de8c047-b0f7-4615-a1ab-5923a2d8141b-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7ddccc4fc8-hfwhh\" (UID: \"2de8c047-b0f7-4615-a1ab-5923a2d8141b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7ddccc4fc8-hfwhh"
Jul 28 10:08:46.487705 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:46.487681    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpt9f\" (UniqueName: \"kubernetes.io/projected/2de8c047-b0f7-4615-a1ab-5923a2d8141b-kube-api-access-gpt9f\") pod \"managed-serviceaccount-addon-agent-7ddccc4fc8-hfwhh\" (UID: \"2de8c047-b0f7-4615-a1ab-5923a2d8141b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7ddccc4fc8-hfwhh"
Jul 28 10:08:46.511839 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:46.511812    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6b5bff677c-7d2rn"]
Jul 28 10:08:46.572775 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:46.572713    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:46.580893 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:46.580872    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-54f7d57974-x8r6k"]
Jul 28 10:08:46.591929 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:46.591906    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54f7d57974-x8r6k"
Jul 28 10:08:46.608327 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:46.608306    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-54f7d57974-x8r6k"]
Jul 28 10:08:46.609493 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:46.609475    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7ddccc4fc8-hfwhh"
Jul 28 10:08:46.657489 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:46.657081    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0a008076-8474-4d60-a865-f058c917827a-trusted-ca\") pod \"image-registry-54f7d57974-x8r6k\" (UID: \"0a008076-8474-4d60-a865-f058c917827a\") " pod="openshift-image-registry/image-registry-54f7d57974-x8r6k"
Jul 28 10:08:46.657489 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:46.657132    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0a008076-8474-4d60-a865-f058c917827a-image-registry-private-configuration\") pod \"image-registry-54f7d57974-x8r6k\" (UID: \"0a008076-8474-4d60-a865-f058c917827a\") " pod="openshift-image-registry/image-registry-54f7d57974-x8r6k"
Jul 28 10:08:46.657489 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:46.657158    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0a008076-8474-4d60-a865-f058c917827a-registry-certificates\") pod \"image-registry-54f7d57974-x8r6k\" (UID: \"0a008076-8474-4d60-a865-f058c917827a\") " pod="openshift-image-registry/image-registry-54f7d57974-x8r6k"
Jul 28 10:08:46.657489 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:46.657233    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0a008076-8474-4d60-a865-f058c917827a-ca-trust-extracted\") pod \"image-registry-54f7d57974-x8r6k\" (UID: \"0a008076-8474-4d60-a865-f058c917827a\") " pod="openshift-image-registry/image-registry-54f7d57974-x8r6k"
Jul 28 10:08:46.657489 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:46.657270    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrxk7\" (UniqueName: \"kubernetes.io/projected/0a008076-8474-4d60-a865-f058c917827a-kube-api-access-zrxk7\") pod \"image-registry-54f7d57974-x8r6k\" (UID: \"0a008076-8474-4d60-a865-f058c917827a\") " pod="openshift-image-registry/image-registry-54f7d57974-x8r6k"
Jul 28 10:08:46.657489 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:46.657306    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0a008076-8474-4d60-a865-f058c917827a-bound-sa-token\") pod \"image-registry-54f7d57974-x8r6k\" (UID: \"0a008076-8474-4d60-a865-f058c917827a\") " pod="openshift-image-registry/image-registry-54f7d57974-x8r6k"
Jul 28 10:08:46.657489 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:46.657354    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0a008076-8474-4d60-a865-f058c917827a-installation-pull-secrets\") pod \"image-registry-54f7d57974-x8r6k\" (UID: \"0a008076-8474-4d60-a865-f058c917827a\") " pod="openshift-image-registry/image-registry-54f7d57974-x8r6k"
Jul 28 10:08:46.657489 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:46.657384    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0a008076-8474-4d60-a865-f058c917827a-registry-tls\") pod \"image-registry-54f7d57974-x8r6k\" (UID: \"0a008076-8474-4d60-a865-f058c917827a\") " pod="openshift-image-registry/image-registry-54f7d57974-x8r6k"
Jul 28 10:08:46.675451 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:46.675398    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:46.757883 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:46.757852    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0a008076-8474-4d60-a865-f058c917827a-registry-tls\") pod \"image-registry-54f7d57974-x8r6k\" (UID: \"0a008076-8474-4d60-a865-f058c917827a\") " pod="openshift-image-registry/image-registry-54f7d57974-x8r6k"
Jul 28 10:08:46.758321 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:46.757910    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0a008076-8474-4d60-a865-f058c917827a-trusted-ca\") pod \"image-registry-54f7d57974-x8r6k\" (UID: \"0a008076-8474-4d60-a865-f058c917827a\") " pod="openshift-image-registry/image-registry-54f7d57974-x8r6k"
Jul 28 10:08:46.758321 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:46.757947    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0a008076-8474-4d60-a865-f058c917827a-image-registry-private-configuration\") pod \"image-registry-54f7d57974-x8r6k\" (UID: \"0a008076-8474-4d60-a865-f058c917827a\") " pod="openshift-image-registry/image-registry-54f7d57974-x8r6k"
Jul 28 10:08:46.758321 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:46.757968    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0a008076-8474-4d60-a865-f058c917827a-registry-certificates\") pod \"image-registry-54f7d57974-x8r6k\" (UID: \"0a008076-8474-4d60-a865-f058c917827a\") " pod="openshift-image-registry/image-registry-54f7d57974-x8r6k"
Jul 28 10:08:46.758321 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:46.758015    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0a008076-8474-4d60-a865-f058c917827a-ca-trust-extracted\") pod \"image-registry-54f7d57974-x8r6k\" (UID: \"0a008076-8474-4d60-a865-f058c917827a\") " pod="openshift-image-registry/image-registry-54f7d57974-x8r6k"
Jul 28 10:08:46.758321 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:46.758052    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrxk7\" (UniqueName: \"kubernetes.io/projected/0a008076-8474-4d60-a865-f058c917827a-kube-api-access-zrxk7\") pod \"image-registry-54f7d57974-x8r6k\" (UID: \"0a008076-8474-4d60-a865-f058c917827a\") " pod="openshift-image-registry/image-registry-54f7d57974-x8r6k"
Jul 28 10:08:46.758321 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:46.758094    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0a008076-8474-4d60-a865-f058c917827a-bound-sa-token\") pod \"image-registry-54f7d57974-x8r6k\" (UID: \"0a008076-8474-4d60-a865-f058c917827a\") " pod="openshift-image-registry/image-registry-54f7d57974-x8r6k"
Jul 28 10:08:46.758321 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:46.758146    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0a008076-8474-4d60-a865-f058c917827a-installation-pull-secrets\") pod \"image-registry-54f7d57974-x8r6k\" (UID: \"0a008076-8474-4d60-a865-f058c917827a\") " pod="openshift-image-registry/image-registry-54f7d57974-x8r6k"
Jul 28 10:08:46.759090 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:46.759017    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0a008076-8474-4d60-a865-f058c917827a-trusted-ca\") pod \"image-registry-54f7d57974-x8r6k\" (UID: \"0a008076-8474-4d60-a865-f058c917827a\") " pod="openshift-image-registry/image-registry-54f7d57974-x8r6k"
Jul 28 10:08:46.759216 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:46.759157    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0a008076-8474-4d60-a865-f058c917827a-ca-trust-extracted\") pod \"image-registry-54f7d57974-x8r6k\" (UID: \"0a008076-8474-4d60-a865-f058c917827a\") " pod="openshift-image-registry/image-registry-54f7d57974-x8r6k"
Jul 28 10:08:46.759427 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:46.759382    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0a008076-8474-4d60-a865-f058c917827a-registry-certificates\") pod \"image-registry-54f7d57974-x8r6k\" (UID: \"0a008076-8474-4d60-a865-f058c917827a\") " pod="openshift-image-registry/image-registry-54f7d57974-x8r6k"
Jul 28 10:08:46.760750 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:46.760726    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0a008076-8474-4d60-a865-f058c917827a-registry-tls\") pod \"image-registry-54f7d57974-x8r6k\" (UID: \"0a008076-8474-4d60-a865-f058c917827a\") " pod="openshift-image-registry/image-registry-54f7d57974-x8r6k"
Jul 28 10:08:46.760844 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:46.760769    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0a008076-8474-4d60-a865-f058c917827a-image-registry-private-configuration\") pod \"image-registry-54f7d57974-x8r6k\" (UID: \"0a008076-8474-4d60-a865-f058c917827a\") " pod="openshift-image-registry/image-registry-54f7d57974-x8r6k"
Jul 28 10:08:46.761322 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:46.761300    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0a008076-8474-4d60-a865-f058c917827a-installation-pull-secrets\") pod \"image-registry-54f7d57974-x8r6k\" (UID: \"0a008076-8474-4d60-a865-f058c917827a\") " pod="openshift-image-registry/image-registry-54f7d57974-x8r6k"
Jul 28 10:08:46.771083 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:46.771062    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7ddccc4fc8-hfwhh"]
Jul 28 10:08:46.777640 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:46.777616    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0a008076-8474-4d60-a865-f058c917827a-bound-sa-token\") pod \"image-registry-54f7d57974-x8r6k\" (UID: \"0a008076-8474-4d60-a865-f058c917827a\") " pod="openshift-image-registry/image-registry-54f7d57974-x8r6k"
Jul 28 10:08:46.778081 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:46.778046    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:46.778341 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:46.778324    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrxk7\" (UniqueName: \"kubernetes.io/projected/0a008076-8474-4d60-a865-f058c917827a-kube-api-access-zrxk7\") pod \"image-registry-54f7d57974-x8r6k\" (UID: \"0a008076-8474-4d60-a865-f058c917827a\") " pod="openshift-image-registry/image-registry-54f7d57974-x8r6k"
Jul 28 10:08:46.902034 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:46.902010    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54f7d57974-x8r6k"
Jul 28 10:08:46.986379 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:46.986352    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:47.059685 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:47.059660    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19db4187-c2f2-46d9-bb21-ce1d3c12c422-metrics-certs\") pod \"network-metrics-daemon-ldq5b\" (UID: \"19db4187-c2f2-46d9-bb21-ce1d3c12c422\") " pod="openshift-multus/network-metrics-daemon-ldq5b"
Jul 28 10:08:47.063112 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:47.063092    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 10:08:47.073146 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:47.073126    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19db4187-c2f2-46d9-bb21-ce1d3c12c422-metrics-certs\") pod \"network-metrics-daemon-ldq5b\" (UID: \"19db4187-c2f2-46d9-bb21-ce1d3c12c422\") " pod="openshift-multus/network-metrics-daemon-ldq5b"
Jul 28 10:08:47.158153 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:47.158097    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:47.160239 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:47.160212    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gzlj\" (UniqueName: \"kubernetes.io/projected/d329552c-d2a3-4960-9130-bc9e211efc18-kube-api-access-2gzlj\") pod \"network-check-target-7dfzm\" (UID: \"d329552c-d2a3-4960-9130-bc9e211efc18\") " pod="openshift-network-diagnostics/network-check-target-7dfzm"
Jul 28 10:08:47.160325 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:47.160261    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/07ad2626-9f01-48ec-adb7-c5f4ab340b7a-original-pull-secret\") pod \"global-pull-secret-syncer-64pm8\" (UID: \"07ad2626-9f01-48ec-adb7-c5f4ab340b7a\") " pod="kube-system/global-pull-secret-syncer-64pm8"
Jul 28 10:08:47.163161 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:47.163142    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 10:08:47.168662 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:47.168637    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 10:08:47.172873 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:47.172854    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/07ad2626-9f01-48ec-adb7-c5f4ab340b7a-original-pull-secret\") pod \"global-pull-secret-syncer-64pm8\" (UID: \"07ad2626-9f01-48ec-adb7-c5f4ab340b7a\") " pod="kube-system/global-pull-secret-syncer-64pm8"
Jul 28 10:08:47.173586 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:47.173564    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 10:08:47.183978 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:47.183959    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gzlj\" (UniqueName: \"kubernetes.io/projected/d329552c-d2a3-4960-9130-bc9e211efc18-kube-api-access-2gzlj\") pod \"network-check-target-7dfzm\" (UID: \"d329552c-d2a3-4960-9130-bc9e211efc18\") " pod="openshift-network-diagnostics/network-check-target-7dfzm"
Jul 28 10:08:47.316447 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:47.316420    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:47.345644 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:47.345616    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-64pm8"
Jul 28 10:08:47.357860 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:47.357834    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rf2fq\""
Jul 28 10:08:47.360765 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:47.360740    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ltg76\""
Jul 28 10:08:47.363556 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:47.363536    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldq5b"
Jul 28 10:08:47.369316 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:47.369294    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7dfzm"
Jul 28 10:08:47.462768 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:47.462702    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:48.337249 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:08:48.337219    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2de8c047_b0f7_4615_a1ab_5923a2d8141b.slice/crio-e09735c602bfe2b5321c6bd71b301a6c1ededbea1f93ec09b65bd888662385be WatchSource:0}: Error finding container e09735c602bfe2b5321c6bd71b301a6c1ededbea1f93ec09b65bd888662385be: Status 404 returned error can't find the container with id e09735c602bfe2b5321c6bd71b301a6c1ededbea1f93ec09b65bd888662385be
Jul 28 10:08:48.508085 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:48.508056    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7dfzm"]
Jul 28 10:08:48.521913 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:08:48.521875    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd329552c_d2a3_4960_9130_bc9e211efc18.slice/crio-3d2d688e5735352ad7e0715248004c9a54b67aa99a214a86c728376476e04588 WatchSource:0}: Error finding container 3d2d688e5735352ad7e0715248004c9a54b67aa99a214a86c728376476e04588: Status 404 returned error can't find the container with id 3d2d688e5735352ad7e0715248004c9a54b67aa99a214a86c728376476e04588
Jul 28 10:08:48.548238 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:48.548216    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ldq5b"]
Jul 28 10:08:48.552071 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:08:48.552046    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19db4187_c2f2_46d9_bb21_ce1d3c12c422.slice/crio-d735057993c0e2413f0d293f4300e7723f354e598c66993ca4718a4027b58082 WatchSource:0}: Error finding container d735057993c0e2413f0d293f4300e7723f354e598c66993ca4718a4027b58082: Status 404 returned error can't find the container with id d735057993c0e2413f0d293f4300e7723f354e598c66993ca4718a4027b58082
Jul 28 10:08:48.705872 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:48.705843    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7dfzm" event={"ID":"d329552c-d2a3-4960-9130-bc9e211efc18","Type":"ContainerStarted","Data":"3d2d688e5735352ad7e0715248004c9a54b67aa99a214a86c728376476e04588"}
Jul 28 10:08:48.708020 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:48.707998    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-h9wkh" event={"ID":"1f82a539-9c16-4566-96ae-7d63e58d46d2","Type":"ContainerStarted","Data":"da352ec55fcefe01a0ccac72ffd9e4757efee7ca3f6bc70da68899f933ae10c2"}
Jul 28 10:08:48.709142 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:48.709116    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7ddccc4fc8-hfwhh" event={"ID":"2de8c047-b0f7-4615-a1ab-5923a2d8141b","Type":"ContainerStarted","Data":"e09735c602bfe2b5321c6bd71b301a6c1ededbea1f93ec09b65bd888662385be"}
Jul 28 10:08:48.710164 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:48.710143    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ldq5b" event={"ID":"19db4187-c2f2-46d9-bb21-ce1d3c12c422","Type":"ContainerStarted","Data":"d735057993c0e2413f0d293f4300e7723f354e598c66993ca4718a4027b58082"}
Jul 28 10:08:48.732235 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:48.732167    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-h9wkh" podStartSLOduration=17.014759109 podStartE2EDuration="29.732152135s" totalImagesPullingTime="12.717393026s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:08:19 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:48.731586282 +0000 UTC m=+67.832092840" watchObservedRunningTime="2026-07-28 10:08:48.732152135 +0000 UTC m=+67.832658707"
Jul 28 10:08:48.745581 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:48.745558    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-64pm8"]
Jul 28 10:08:48.748087 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:08:48.748058    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07ad2626_9f01_48ec_adb7_c5f4ab340b7a.slice/crio-b9978bb044b5d42656f3c1cef29f8999edde04597b08396edc731d6e8a8821b2 WatchSource:0}: Error finding container b9978bb044b5d42656f3c1cef29f8999edde04597b08396edc731d6e8a8821b2: Status 404 returned error can't find the container with id b9978bb044b5d42656f3c1cef29f8999edde04597b08396edc731d6e8a8821b2
Jul 28 10:08:48.755802 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:48.755784    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-54f7d57974-x8r6k"]
Jul 28 10:08:48.757945 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:08:48.757923    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a008076_8474_4d60_a865_f058c917827a.slice/crio-1ed8859d6a1908995c10d4f530c24628d09553d7d635a55a19f4d9777972601d WatchSource:0}: Error finding container 1ed8859d6a1908995c10d4f530c24628d09553d7d635a55a19f4d9777972601d: Status 404 returned error can't find the container with id 1ed8859d6a1908995c10d4f530c24628d09553d7d635a55a19f4d9777972601d
Jul 28 10:08:49.717643 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:49.717604    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54f7d57974-x8r6k" event={"ID":"0a008076-8474-4d60-a865-f058c917827a","Type":"ContainerStarted","Data":"09e559581ca0ee23030d20ec2ed7fd97d9d4f3f1e1c0b1460ac48e7f65cc644e"}
Jul 28 10:08:49.718108 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:49.717671    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54f7d57974-x8r6k" event={"ID":"0a008076-8474-4d60-a865-f058c917827a","Type":"ContainerStarted","Data":"1ed8859d6a1908995c10d4f530c24628d09553d7d635a55a19f4d9777972601d"}
Jul 28 10:08:49.718462 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:49.718441    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-54f7d57974-x8r6k"
Jul 28 10:08:49.721502 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:49.721468    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-64pm8" event={"ID":"07ad2626-9f01-48ec-adb7-c5f4ab340b7a","Type":"ContainerStarted","Data":"b9978bb044b5d42656f3c1cef29f8999edde04597b08396edc731d6e8a8821b2"}
Jul 28 10:08:49.747041 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:49.746685    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-54f7d57974-x8r6k" podStartSLOduration=3.746667326 podStartE2EDuration="3.746667326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:08:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:49.744485588 +0000 UTC m=+68.844992142" watchObservedRunningTime="2026-07-28 10:08:49.746667326 +0000 UTC m=+68.847173882"
Jul 28 10:08:50.552824 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:50.552784    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:50.689716 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:50.689679    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f45e9c0a-7761-4581-8ae8-dd652ec7846f-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-ccv4b\" (UID: \"f45e9c0a-7761-4581-8ae8-dd652ec7846f\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-ccv4b"
Jul 28 10:08:50.701860 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:50.695569    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:50.701860 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:50.695950    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f45e9c0a-7761-4581-8ae8-dd652ec7846f-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-ccv4b\" (UID: \"f45e9c0a-7761-4581-8ae8-dd652ec7846f\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-ccv4b"
Jul 28 10:08:50.729664 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:50.729630    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ldq5b" event={"ID":"19db4187-c2f2-46d9-bb21-ce1d3c12c422","Type":"ContainerStarted","Data":"4aebad328e0193c613978758d25aacd0e761992ca062b0c81abbd17949f1361e"}
Jul 28 10:08:50.730166 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:50.729695    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ldq5b" event={"ID":"19db4187-c2f2-46d9-bb21-ce1d3c12c422","Type":"ContainerStarted","Data":"3b84d0ca2402187657d68e71f001a7e95fe195f122ee6aa31abca330b071868d"}
Jul 28 10:08:50.770219 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:50.770155    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-ldq5b" podStartSLOduration=68.399681949 podStartE2EDuration="1m9.770138014s" totalImagesPullingTime="1.370456065s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:07:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:50.769607053 +0000 UTC m=+69.870113632" watchObservedRunningTime="2026-07-28 10:08:50.770138014 +0000 UTC m=+69.870644567"
Jul 28 10:08:50.824435 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:50.824361    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:50.912333 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:50.912298    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:50.955086 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:50.955056    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-vzfmh\""
Jul 28 10:08:50.957124 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:50.957094    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-ccv4b"
Jul 28 10:08:50.982655 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:50.982622    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:51.124525 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:51.124443    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:51.183667 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:51.183633    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:52.737389 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:52.737352    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:52.838118 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:52.838090    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:53.163528 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:53.163485    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:53.620696 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:53.620619    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:53.704371 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:53.704340    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:54.460621 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:54.460562    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-ccv4b"]
Jul 28 10:08:54.744604 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:54.744571    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-64pm8" event={"ID":"07ad2626-9f01-48ec-adb7-c5f4ab340b7a","Type":"ContainerStarted","Data":"719b489b94e0fe2a7e6b040dd1a79a635dcd5f74cc157024a3bd7b0c5b604eab"}
Jul 28 10:08:54.746163 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:54.746134    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7ddccc4fc8-hfwhh" event={"ID":"2de8c047-b0f7-4615-a1ab-5923a2d8141b","Type":"ContainerStarted","Data":"8ecfbdccbc15e34d464caeeba98b11e2fd785dd325ef2823e696da998af1f244"}
Jul 28 10:08:54.747241 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:54.747213    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-ccv4b" event={"ID":"f45e9c0a-7761-4581-8ae8-dd652ec7846f","Type":"ContainerStarted","Data":"b8f77e6dedd4a37b55af3dca8dd3ce0a1ff88054e0b9c701482b91ec782c4845"}
Jul 28 10:08:54.748442 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:54.748418    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7dfzm" event={"ID":"d329552c-d2a3-4960-9130-bc9e211efc18","Type":"ContainerStarted","Data":"53ef080563eec5cefa19876c58f2e9ed4bb5a4c74c3a7f2f1a2a11248d097b76"}
Jul 28 10:08:54.748574 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:54.748556    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-7dfzm"
Jul 28 10:08:54.767701 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:54.767658    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-64pm8" podStartSLOduration=68.186768272 podStartE2EDuration="1m13.767644575s" totalImagesPullingTime="5.580876303s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:07:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:54.766324783 +0000 UTC m=+73.866831328" watchObservedRunningTime="2026-07-28 10:08:54.767644575 +0000 UTC m=+73.868151129"
Jul 28 10:08:54.786269 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:54.786231    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7ddccc4fc8-hfwhh" podStartSLOduration=2.808974676 podStartE2EDuration="8.786214993s" totalImagesPullingTime="5.977240317s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:08:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:54.785742104 +0000 UTC m=+73.886248659" watchObservedRunningTime="2026-07-28 10:08:54.786214993 +0000 UTC m=+73.886721549"
Jul 28 10:08:54.810666 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:54.810630    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-7dfzm" podStartSLOduration=68.006655046 podStartE2EDuration="1m13.81061977s" totalImagesPullingTime="5.803964724s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:07:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:54.80899248 +0000 UTC m=+73.909499034" watchObservedRunningTime="2026-07-28 10:08:54.81061977 +0000 UTC m=+73.911126334"
Jul 28 10:08:55.472281 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:55.472246    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:55.540163 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:55.540128    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:55.847652 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:55.847623    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:56.518351 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:56.518326    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6b5bff677c-7d2rn"
Jul 28 10:08:56.757500 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:56.757461    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-ccv4b" event={"ID":"f45e9c0a-7761-4581-8ae8-dd652ec7846f","Type":"ContainerStarted","Data":"4f9e5e3d4e4284d1e74ff29fc0eff791532d0ea9b6acbaa6cf971791f25f5f44"}
Jul 28 10:08:56.778099 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:56.777947    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-ccv4b" podStartSLOduration=37.54442821 podStartE2EDuration="38.777931066s" totalImagesPullingTime="1.233502856s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:08:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:56.777848885 +0000 UTC m=+75.878355439" watchObservedRunningTime="2026-07-28 10:08:56.777931066 +0000 UTC m=+75.878437620"
Jul 28 10:08:56.956314 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:56.956286    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:57.056281 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:57.056215    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:57.898476 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:57.898444    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:58.096132 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.096105    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:58.197591 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.197536    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:58.219353 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.219328    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-d8vg4"]
Jul 28 10:08:58.221913 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.221896    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-d8vg4"
Jul 28 10:08:58.224638 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.224616    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 28 10:08:58.224966 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.224948    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 28 10:08:58.225045 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.224953    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 28 10:08:58.225294 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.225278    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 28 10:08:58.225380 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.225278    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 28 10:08:58.225380 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.225315    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-58cxb\""
Jul 28 10:08:58.235916 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.235899    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-wt5j9"]
Jul 28 10:08:58.238620 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.238603    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-wt5j9"
Jul 28 10:08:58.240865 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.240847    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 28 10:08:58.242401 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.240871    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-r5tr4\""
Jul 28 10:08:58.242401 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.241079    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 28 10:08:58.242401 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.241405    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 28 10:08:58.242944 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.242921    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-d8vg4"]
Jul 28 10:08:58.267825 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.267803    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-wt5j9"]
Jul 28 10:08:58.285460 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.285430    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-d8tr2"]
Jul 28 10:08:58.287941 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.287926    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-d8tr2"
Jul 28 10:08:58.291455 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.291439    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 28 10:08:58.291600 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.291578    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 28 10:08:58.291686 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.291657    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-79tkk\""
Jul 28 10:08:58.291747 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.291695    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 28 10:08:58.315384 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.315363    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:58.346079 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.346053    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8c69d312-24ea-47e7-8f36-1e6a5362010a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-wt5j9\" (UID: \"8c69d312-24ea-47e7-8f36-1e6a5362010a\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-wt5j9"
Jul 28 10:08:58.346171 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.346089    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c69d312-24ea-47e7-8f36-1e6a5362010a-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-wt5j9\" (UID: \"8c69d312-24ea-47e7-8f36-1e6a5362010a\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-wt5j9"
Jul 28 10:08:58.346171 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.346119    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/73013d79-268e-42f8-8bbd-9638f7b29d7a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-d8vg4\" (UID: \"73013d79-268e-42f8-8bbd-9638f7b29d7a\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-d8vg4"
Jul 28 10:08:58.346171 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.346153    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8c69d312-24ea-47e7-8f36-1e6a5362010a-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-wt5j9\" (UID: \"8c69d312-24ea-47e7-8f36-1e6a5362010a\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-wt5j9"
Jul 28 10:08:58.346365 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.346196    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1b50dc93-37d8-40ad-a0c3-f3c36cc72aca-node-exporter-accelerators-collector-config\") pod \"node-exporter-d8tr2\" (UID: \"1b50dc93-37d8-40ad-a0c3-f3c36cc72aca\") " pod="openshift-monitoring/node-exporter-d8tr2"
Jul 28 10:08:58.346365 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.346223    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/73013d79-268e-42f8-8bbd-9638f7b29d7a-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-d8vg4\" (UID: \"73013d79-268e-42f8-8bbd-9638f7b29d7a\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-d8vg4"
Jul 28 10:08:58.346365 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.346247    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1b50dc93-37d8-40ad-a0c3-f3c36cc72aca-sys\") pod \"node-exporter-d8tr2\" (UID: \"1b50dc93-37d8-40ad-a0c3-f3c36cc72aca\") " pod="openshift-monitoring/node-exporter-d8tr2"
Jul 28 10:08:58.346365 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.346270    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1b50dc93-37d8-40ad-a0c3-f3c36cc72aca-node-exporter-wtmp\") pod \"node-exporter-d8tr2\" (UID: \"1b50dc93-37d8-40ad-a0c3-f3c36cc72aca\") " pod="openshift-monitoring/node-exporter-d8tr2"
Jul 28 10:08:58.346365 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.346300    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg66d\" (UniqueName: \"kubernetes.io/projected/8c69d312-24ea-47e7-8f36-1e6a5362010a-kube-api-access-jg66d\") pod \"kube-state-metrics-56d9f944b4-wt5j9\" (UID: \"8c69d312-24ea-47e7-8f36-1e6a5362010a\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-wt5j9"
Jul 28 10:08:58.346365 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.346355    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vtvn\" (UniqueName: \"kubernetes.io/projected/73013d79-268e-42f8-8bbd-9638f7b29d7a-kube-api-access-9vtvn\") pod \"openshift-state-metrics-b8f49f5c6-d8vg4\" (UID: \"73013d79-268e-42f8-8bbd-9638f7b29d7a\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-d8vg4"
Jul 28 10:08:58.346622 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.346380    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b50dc93-37d8-40ad-a0c3-f3c36cc72aca-metrics-client-ca\") pod \"node-exporter-d8tr2\" (UID: \"1b50dc93-37d8-40ad-a0c3-f3c36cc72aca\") " pod="openshift-monitoring/node-exporter-d8tr2"
Jul 28 10:08:58.346622 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.346410    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1b50dc93-37d8-40ad-a0c3-f3c36cc72aca-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-d8tr2\" (UID: \"1b50dc93-37d8-40ad-a0c3-f3c36cc72aca\") " pod="openshift-monitoring/node-exporter-d8tr2"
Jul 28 10:08:58.346622 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.346456    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctpbv\" (UniqueName: \"kubernetes.io/projected/1b50dc93-37d8-40ad-a0c3-f3c36cc72aca-kube-api-access-ctpbv\") pod \"node-exporter-d8tr2\" (UID: \"1b50dc93-37d8-40ad-a0c3-f3c36cc72aca\") " pod="openshift-monitoring/node-exporter-d8tr2"
Jul 28 10:08:58.346622 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.346504    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/73013d79-268e-42f8-8bbd-9638f7b29d7a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-d8vg4\" (UID: \"73013d79-268e-42f8-8bbd-9638f7b29d7a\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-d8vg4"
Jul 28 10:08:58.346622 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.346541    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c69d312-24ea-47e7-8f36-1e6a5362010a-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-wt5j9\" (UID: \"8c69d312-24ea-47e7-8f36-1e6a5362010a\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-wt5j9"
Jul 28 10:08:58.346622 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.346567    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1b50dc93-37d8-40ad-a0c3-f3c36cc72aca-node-exporter-textfile\") pod \"node-exporter-d8tr2\" (UID: \"1b50dc93-37d8-40ad-a0c3-f3c36cc72aca\") " pod="openshift-monitoring/node-exporter-d8tr2"
Jul 28 10:08:58.346837 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.346651    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1b50dc93-37d8-40ad-a0c3-f3c36cc72aca-root\") pod \"node-exporter-d8tr2\" (UID: \"1b50dc93-37d8-40ad-a0c3-f3c36cc72aca\") " pod="openshift-monitoring/node-exporter-d8tr2"
Jul 28 10:08:58.346837 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.346688    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8c69d312-24ea-47e7-8f36-1e6a5362010a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-wt5j9\" (UID: \"8c69d312-24ea-47e7-8f36-1e6a5362010a\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-wt5j9"
Jul 28 10:08:58.346837 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.346716    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1b50dc93-37d8-40ad-a0c3-f3c36cc72aca-node-exporter-tls\") pod \"node-exporter-d8tr2\" (UID: \"1b50dc93-37d8-40ad-a0c3-f3c36cc72aca\") " pod="openshift-monitoring/node-exporter-d8tr2"
Jul 28 10:08:58.447683 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.447632    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vtvn\" (UniqueName: \"kubernetes.io/projected/73013d79-268e-42f8-8bbd-9638f7b29d7a-kube-api-access-9vtvn\") pod \"openshift-state-metrics-b8f49f5c6-d8vg4\" (UID: \"73013d79-268e-42f8-8bbd-9638f7b29d7a\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-d8vg4"
Jul 28 10:08:58.447767 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.447659    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b50dc93-37d8-40ad-a0c3-f3c36cc72aca-metrics-client-ca\") pod \"node-exporter-d8tr2\" (UID: \"1b50dc93-37d8-40ad-a0c3-f3c36cc72aca\") " pod="openshift-monitoring/node-exporter-d8tr2"
Jul 28 10:08:58.447767 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.447712    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1b50dc93-37d8-40ad-a0c3-f3c36cc72aca-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-d8tr2\" (UID: \"1b50dc93-37d8-40ad-a0c3-f3c36cc72aca\") " pod="openshift-monitoring/node-exporter-d8tr2"
Jul 28 10:08:58.447767 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.447742    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctpbv\" (UniqueName: \"kubernetes.io/projected/1b50dc93-37d8-40ad-a0c3-f3c36cc72aca-kube-api-access-ctpbv\") pod \"node-exporter-d8tr2\" (UID: \"1b50dc93-37d8-40ad-a0c3-f3c36cc72aca\") " pod="openshift-monitoring/node-exporter-d8tr2"
Jul 28 10:08:58.447925 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.447770    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/73013d79-268e-42f8-8bbd-9638f7b29d7a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-d8vg4\" (UID: \"73013d79-268e-42f8-8bbd-9638f7b29d7a\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-d8vg4"
Jul 28 10:08:58.447925 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.447802    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c69d312-24ea-47e7-8f36-1e6a5362010a-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-wt5j9\" (UID: \"8c69d312-24ea-47e7-8f36-1e6a5362010a\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-wt5j9"
Jul 28 10:08:58.447925 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.447825    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1b50dc93-37d8-40ad-a0c3-f3c36cc72aca-node-exporter-textfile\") pod \"node-exporter-d8tr2\" (UID: \"1b50dc93-37d8-40ad-a0c3-f3c36cc72aca\") " pod="openshift-monitoring/node-exporter-d8tr2"
Jul 28 10:08:58.447925 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.447854    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1b50dc93-37d8-40ad-a0c3-f3c36cc72aca-root\") pod \"node-exporter-d8tr2\" (UID: \"1b50dc93-37d8-40ad-a0c3-f3c36cc72aca\") " pod="openshift-monitoring/node-exporter-d8tr2"
Jul 28 10:08:58.447925 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.447886    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8c69d312-24ea-47e7-8f36-1e6a5362010a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-wt5j9\" (UID: \"8c69d312-24ea-47e7-8f36-1e6a5362010a\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-wt5j9"
Jul 28 10:08:58.448241 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.447946    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1b50dc93-37d8-40ad-a0c3-f3c36cc72aca-root\") pod \"node-exporter-d8tr2\" (UID: \"1b50dc93-37d8-40ad-a0c3-f3c36cc72aca\") " pod="openshift-monitoring/node-exporter-d8tr2"
Jul 28 10:08:58.448241 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.447989    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1b50dc93-37d8-40ad-a0c3-f3c36cc72aca-node-exporter-tls\") pod \"node-exporter-d8tr2\" (UID: \"1b50dc93-37d8-40ad-a0c3-f3c36cc72aca\") " pod="openshift-monitoring/node-exporter-d8tr2"
Jul 28 10:08:58.448241 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.448028    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8c69d312-24ea-47e7-8f36-1e6a5362010a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-wt5j9\" (UID: \"8c69d312-24ea-47e7-8f36-1e6a5362010a\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-wt5j9"
Jul 28 10:08:58.448241 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.448055    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c69d312-24ea-47e7-8f36-1e6a5362010a-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-wt5j9\" (UID: \"8c69d312-24ea-47e7-8f36-1e6a5362010a\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-wt5j9"
Jul 28 10:08:58.448241 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.448085    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/73013d79-268e-42f8-8bbd-9638f7b29d7a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-d8vg4\" (UID: \"73013d79-268e-42f8-8bbd-9638f7b29d7a\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-d8vg4"
Jul 28 10:08:58.448241 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.448114    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8c69d312-24ea-47e7-8f36-1e6a5362010a-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-wt5j9\" (UID: \"8c69d312-24ea-47e7-8f36-1e6a5362010a\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-wt5j9"
Jul 28 10:08:58.448241 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.448149    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1b50dc93-37d8-40ad-a0c3-f3c36cc72aca-node-exporter-accelerators-collector-config\") pod \"node-exporter-d8tr2\" (UID: \"1b50dc93-37d8-40ad-a0c3-f3c36cc72aca\") " pod="openshift-monitoring/node-exporter-d8tr2"
Jul 28 10:08:58.448241 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.448173    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/73013d79-268e-42f8-8bbd-9638f7b29d7a-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-d8vg4\" (UID: \"73013d79-268e-42f8-8bbd-9638f7b29d7a\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-d8vg4"
Jul 28 10:08:58.448241 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.448213    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1b50dc93-37d8-40ad-a0c3-f3c36cc72aca-sys\") pod \"node-exporter-d8tr2\" (UID: \"1b50dc93-37d8-40ad-a0c3-f3c36cc72aca\") " pod="openshift-monitoring/node-exporter-d8tr2"
Jul 28 10:08:58.448241 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.448239    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1b50dc93-37d8-40ad-a0c3-f3c36cc72aca-node-exporter-wtmp\") pod \"node-exporter-d8tr2\" (UID: \"1b50dc93-37d8-40ad-a0c3-f3c36cc72aca\") " pod="openshift-monitoring/node-exporter-d8tr2"
Jul 28 10:08:58.448734 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.448251    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1b50dc93-37d8-40ad-a0c3-f3c36cc72aca-node-exporter-textfile\") pod \"node-exporter-d8tr2\" (UID: \"1b50dc93-37d8-40ad-a0c3-f3c36cc72aca\") " pod="openshift-monitoring/node-exporter-d8tr2"
Jul 28 10:08:58.448734 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.448274    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg66d\" (UniqueName: \"kubernetes.io/projected/8c69d312-24ea-47e7-8f36-1e6a5362010a-kube-api-access-jg66d\") pod \"kube-state-metrics-56d9f944b4-wt5j9\" (UID: \"8c69d312-24ea-47e7-8f36-1e6a5362010a\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-wt5j9"
Jul 28 10:08:58.448734 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.448565    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b50dc93-37d8-40ad-a0c3-f3c36cc72aca-metrics-client-ca\") pod \"node-exporter-d8tr2\" (UID: \"1b50dc93-37d8-40ad-a0c3-f3c36cc72aca\") " pod="openshift-monitoring/node-exporter-d8tr2"
Jul 28 10:08:58.448734 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:58.448621    2583 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jul 28 10:08:58.448734 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.448657    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1b50dc93-37d8-40ad-a0c3-f3c36cc72aca-sys\") pod \"node-exporter-d8tr2\" (UID: \"1b50dc93-37d8-40ad-a0c3-f3c36cc72aca\") " pod="openshift-monitoring/node-exporter-d8tr2"
Jul 28 10:08:58.448734 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:58.448679    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c69d312-24ea-47e7-8f36-1e6a5362010a-kube-state-metrics-tls podName:8c69d312-24ea-47e7-8f36-1e6a5362010a nodeName:}" failed. No retries permitted until 2026-07-28 10:08:58.948658661 +0000 UTC m=+78.049165207 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/8c69d312-24ea-47e7-8f36-1e6a5362010a-kube-state-metrics-tls") pod "kube-state-metrics-56d9f944b4-wt5j9" (UID: "8c69d312-24ea-47e7-8f36-1e6a5362010a") : secret "kube-state-metrics-tls" not found
Jul 28 10:08:58.449011 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.448818    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8c69d312-24ea-47e7-8f36-1e6a5362010a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-wt5j9\" (UID: \"8c69d312-24ea-47e7-8f36-1e6a5362010a\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-wt5j9"
Jul 28 10:08:58.449011 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.448948    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1b50dc93-37d8-40ad-a0c3-f3c36cc72aca-node-exporter-accelerators-collector-config\") pod \"node-exporter-d8tr2\" (UID: \"1b50dc93-37d8-40ad-a0c3-f3c36cc72aca\") " pod="openshift-monitoring/node-exporter-d8tr2"
Jul 28 10:08:58.449011 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.448965    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1b50dc93-37d8-40ad-a0c3-f3c36cc72aca-node-exporter-wtmp\") pod \"node-exporter-d8tr2\" (UID: \"1b50dc93-37d8-40ad-a0c3-f3c36cc72aca\") " pod="openshift-monitoring/node-exporter-d8tr2"
Jul 28 10:08:58.449106 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.449042    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8c69d312-24ea-47e7-8f36-1e6a5362010a-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-wt5j9\" (UID: \"8c69d312-24ea-47e7-8f36-1e6a5362010a\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-wt5j9"
Jul 28 10:08:58.449280 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.449259    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/73013d79-268e-42f8-8bbd-9638f7b29d7a-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-d8vg4\" (UID: \"73013d79-268e-42f8-8bbd-9638f7b29d7a\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-d8vg4"
Jul 28 10:08:58.449651 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.449629    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c69d312-24ea-47e7-8f36-1e6a5362010a-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-wt5j9\" (UID: \"8c69d312-24ea-47e7-8f36-1e6a5362010a\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-wt5j9"
Jul 28 10:08:58.450804 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.450779    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1b50dc93-37d8-40ad-a0c3-f3c36cc72aca-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-d8tr2\" (UID: \"1b50dc93-37d8-40ad-a0c3-f3c36cc72aca\") " pod="openshift-monitoring/node-exporter-d8tr2"
Jul 28 10:08:58.450889 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.450786    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8c69d312-24ea-47e7-8f36-1e6a5362010a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-wt5j9\" (UID: \"8c69d312-24ea-47e7-8f36-1e6a5362010a\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-wt5j9"
Jul 28 10:08:58.450889 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.450877    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/73013d79-268e-42f8-8bbd-9638f7b29d7a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-d8vg4\" (UID: \"73013d79-268e-42f8-8bbd-9638f7b29d7a\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-d8vg4"
Jul 28 10:08:58.450962 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.450895    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1b50dc93-37d8-40ad-a0c3-f3c36cc72aca-node-exporter-tls\") pod \"node-exporter-d8tr2\" (UID: \"1b50dc93-37d8-40ad-a0c3-f3c36cc72aca\") " pod="openshift-monitoring/node-exporter-d8tr2"
Jul 28 10:08:58.451570 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.451549    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/73013d79-268e-42f8-8bbd-9638f7b29d7a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-d8vg4\" (UID: \"73013d79-268e-42f8-8bbd-9638f7b29d7a\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-d8vg4"
Jul 28 10:08:58.477196 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.476635    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctpbv\" (UniqueName: \"kubernetes.io/projected/1b50dc93-37d8-40ad-a0c3-f3c36cc72aca-kube-api-access-ctpbv\") pod \"node-exporter-d8tr2\" (UID: \"1b50dc93-37d8-40ad-a0c3-f3c36cc72aca\") " pod="openshift-monitoring/node-exporter-d8tr2"
Jul 28 10:08:58.477196 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.476752    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg66d\" (UniqueName: \"kubernetes.io/projected/8c69d312-24ea-47e7-8f36-1e6a5362010a-kube-api-access-jg66d\") pod \"kube-state-metrics-56d9f944b4-wt5j9\" (UID: \"8c69d312-24ea-47e7-8f36-1e6a5362010a\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-wt5j9"
Jul 28 10:08:58.477196 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.476809    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vtvn\" (UniqueName: \"kubernetes.io/projected/73013d79-268e-42f8-8bbd-9638f7b29d7a-kube-api-access-9vtvn\") pod \"openshift-state-metrics-b8f49f5c6-d8vg4\" (UID: \"73013d79-268e-42f8-8bbd-9638f7b29d7a\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-d8vg4"
Jul 28 10:08:58.493350 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.493320    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:58.531400 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.531379    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-d8vg4"
Jul 28 10:08:58.596543 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.596291    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-d8tr2"
Jul 28 10:08:58.599358 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.599318    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:58.609447 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:08:58.609426    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b50dc93_37d8_40ad_a0c3_f3c36cc72aca.slice/crio-e83b8be26baece52dbe174828fa16672942df680f4da8d3aea855036f699fdd8 WatchSource:0}: Error finding container e83b8be26baece52dbe174828fa16672942df680f4da8d3aea855036f699fdd8: Status 404 returned error can't find the container with id e83b8be26baece52dbe174828fa16672942df680f4da8d3aea855036f699fdd8
Jul 28 10:08:58.673710 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.673681    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-d8vg4"]
Jul 28 10:08:58.676156 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:08:58.676129    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73013d79_268e_42f8_8bbd_9638f7b29d7a.slice/crio-edf84f8225f6a990fa59b7ad48ab6c9857f4d8decc9f1a6ae812062178056d8a WatchSource:0}: Error finding container edf84f8225f6a990fa59b7ad48ab6c9857f4d8decc9f1a6ae812062178056d8a: Status 404 returned error can't find the container with id edf84f8225f6a990fa59b7ad48ab6c9857f4d8decc9f1a6ae812062178056d8a
Jul 28 10:08:58.676535 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.676515    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:58.766303 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.766272    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d8tr2" event={"ID":"1b50dc93-37d8-40ad-a0c3-f3c36cc72aca","Type":"ContainerStarted","Data":"e83b8be26baece52dbe174828fa16672942df680f4da8d3aea855036f699fdd8"}
Jul 28 10:08:58.767763 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.767742    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-d8vg4" event={"ID":"73013d79-268e-42f8-8bbd-9638f7b29d7a","Type":"ContainerStarted","Data":"f8baf2c644d1c507a328e64b67b779c2204c95bec79f0453e7bc071eb809a27a"}
Jul 28 10:08:58.767852 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.767786    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-d8vg4" event={"ID":"73013d79-268e-42f8-8bbd-9638f7b29d7a","Type":"ContainerStarted","Data":"edf84f8225f6a990fa59b7ad48ab6c9857f4d8decc9f1a6ae812062178056d8a"}
Jul 28 10:08:58.777844 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.777819    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:58.951994 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.951962    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c69d312-24ea-47e7-8f36-1e6a5362010a-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-wt5j9\" (UID: \"8c69d312-24ea-47e7-8f36-1e6a5362010a\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-wt5j9"
Jul 28 10:08:58.954303 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:58.954281    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c69d312-24ea-47e7-8f36-1e6a5362010a-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-wt5j9\" (UID: \"8c69d312-24ea-47e7-8f36-1e6a5362010a\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-wt5j9"
Jul 28 10:08:59.149200 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:59.149155    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-wt5j9"
Jul 28 10:08:59.156603 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:59.156583    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:59.264495 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:59.264464    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 10:08:59.270402 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:59.269719    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:08:59.274808 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:59.274768    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 28 10:08:59.275201 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:59.275149    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 28 10:08:59.275307 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:59.275206    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 28 10:08:59.275385 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:59.275325    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 28 10:08:59.275452 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:59.275432    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 28 10:08:59.275570 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:59.275493    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 28 10:08:59.275570 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:59.275496    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 28 10:08:59.275776 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:59.275723    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 28 10:08:59.275776 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:59.275740    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 28 10:08:59.275988 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:59.275971    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-pbz2x\""
Jul 28 10:08:59.290114 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:59.290094    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 10:08:59.320555 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:59.320482    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-wt5j9"]
Jul 28 10:08:59.323671 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:08:59.323647    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c69d312_24ea_47e7_8f36_1e6a5362010a.slice/crio-ffd843f16d559795a87d7c232d4832488b507ed5122d63e4c860888508e1dd50 WatchSource:0}: Error finding container ffd843f16d559795a87d7c232d4832488b507ed5122d63e4c860888508e1dd50: Status 404 returned error can't find the container with id ffd843f16d559795a87d7c232d4832488b507ed5122d63e4c860888508e1dd50
Jul 28 10:08:59.348243 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:59.348220    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:59.356534 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:59.356508    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bc4b0f8e-790d-412e-99fc-bef89769970a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"bc4b0f8e-790d-412e-99fc-bef89769970a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:08:59.356642 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:59.356550    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc4b0f8e-790d-412e-99fc-bef89769970a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"bc4b0f8e-790d-412e-99fc-bef89769970a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:08:59.356642 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:59.356580    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bc4b0f8e-790d-412e-99fc-bef89769970a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"bc4b0f8e-790d-412e-99fc-bef89769970a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:08:59.356764 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:59.356743    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bc4b0f8e-790d-412e-99fc-bef89769970a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"bc4b0f8e-790d-412e-99fc-bef89769970a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:08:59.356842 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:59.356827    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bc4b0f8e-790d-412e-99fc-bef89769970a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"bc4b0f8e-790d-412e-99fc-bef89769970a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:08:59.356898 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:59.356861    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/bc4b0f8e-790d-412e-99fc-bef89769970a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"bc4b0f8e-790d-412e-99fc-bef89769970a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:08:59.356951 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:59.356911    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bc4b0f8e-790d-412e-99fc-bef89769970a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"bc4b0f8e-790d-412e-99fc-bef89769970a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:08:59.356992 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:59.356982    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bc4b0f8e-790d-412e-99fc-bef89769970a-config-volume\") pod \"alertmanager-main-0\" (UID: \"bc4b0f8e-790d-412e-99fc-bef89769970a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:08:59.357029 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:59.357007    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx8tf\" (UniqueName: \"kubernetes.io/projected/bc4b0f8e-790d-412e-99fc-bef89769970a-kube-api-access-sx8tf\") pod \"alertmanager-main-0\" (UID: \"bc4b0f8e-790d-412e-99fc-bef89769970a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:08:59.357068 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:59.357036    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/bc4b0f8e-790d-412e-99fc-bef89769970a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"bc4b0f8e-790d-412e-99fc-bef89769970a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:08:59.357121 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:59.357066    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bc4b0f8e-790d-412e-99fc-bef89769970a-config-out\") pod \"alertmanager-main-0\" (UID: \"bc4b0f8e-790d-412e-99fc-bef89769970a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:08:59.357121 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:59.357091    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bc4b0f8e-790d-412e-99fc-bef89769970a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"bc4b0f8e-790d-412e-99fc-bef89769970a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:08:59.357243 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:59.357117    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bc4b0f8e-790d-412e-99fc-bef89769970a-web-config\") pod \"alertmanager-main-0\" (UID: \"bc4b0f8e-790d-412e-99fc-bef89769970a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:08:59.457725 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:59.457661    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bc4b0f8e-790d-412e-99fc-bef89769970a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"bc4b0f8e-790d-412e-99fc-bef89769970a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:08:59.457725 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:59.457702    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bc4b0f8e-790d-412e-99fc-bef89769970a-config-volume\") pod \"alertmanager-main-0\" (UID: \"bc4b0f8e-790d-412e-99fc-bef89769970a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:08:59.457922 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:59.457728    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx8tf\" (UniqueName: \"kubernetes.io/projected/bc4b0f8e-790d-412e-99fc-bef89769970a-kube-api-access-sx8tf\") pod \"alertmanager-main-0\" (UID: \"bc4b0f8e-790d-412e-99fc-bef89769970a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:08:59.457922 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:59.457754    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/bc4b0f8e-790d-412e-99fc-bef89769970a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"bc4b0f8e-790d-412e-99fc-bef89769970a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:08:59.457922 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:59.457780    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bc4b0f8e-790d-412e-99fc-bef89769970a-config-out\") pod \"alertmanager-main-0\" (UID: \"bc4b0f8e-790d-412e-99fc-bef89769970a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:08:59.457922 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:59.457808    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bc4b0f8e-790d-412e-99fc-bef89769970a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"bc4b0f8e-790d-412e-99fc-bef89769970a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:08:59.457922 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:59.457831    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bc4b0f8e-790d-412e-99fc-bef89769970a-web-config\") pod \"alertmanager-main-0\" (UID: \"bc4b0f8e-790d-412e-99fc-bef89769970a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:08:59.457922 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:59.457870    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bc4b0f8e-790d-412e-99fc-bef89769970a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"bc4b0f8e-790d-412e-99fc-bef89769970a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:08:59.457922 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:59.457894    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc4b0f8e-790d-412e-99fc-bef89769970a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"bc4b0f8e-790d-412e-99fc-bef89769970a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:08:59.458305 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:59.457923    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bc4b0f8e-790d-412e-99fc-bef89769970a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"bc4b0f8e-790d-412e-99fc-bef89769970a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:08:59.458305 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:59.457974    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bc4b0f8e-790d-412e-99fc-bef89769970a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"bc4b0f8e-790d-412e-99fc-bef89769970a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:08:59.458305 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:59.458028    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bc4b0f8e-790d-412e-99fc-bef89769970a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"bc4b0f8e-790d-412e-99fc-bef89769970a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:08:59.458305 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:59.458058    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/bc4b0f8e-790d-412e-99fc-bef89769970a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"bc4b0f8e-790d-412e-99fc-bef89769970a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:08:59.458898 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:59.458821    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/bc4b0f8e-790d-412e-99fc-bef89769970a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"bc4b0f8e-790d-412e-99fc-bef89769970a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:08:59.459567 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:08:59.459200    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bc4b0f8e-790d-412e-99fc-bef89769970a-alertmanager-trusted-ca-bundle podName:bc4b0f8e-790d-412e-99fc-bef89769970a nodeName:}" failed. No retries permitted until 2026-07-28 10:08:59.959160301 +0000 UTC m=+79.059666833 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/bc4b0f8e-790d-412e-99fc-bef89769970a-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "bc4b0f8e-790d-412e-99fc-bef89769970a") : configmap references non-existent config key: ca-bundle.crt
Jul 28 10:08:59.459567 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:59.459278    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bc4b0f8e-790d-412e-99fc-bef89769970a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"bc4b0f8e-790d-412e-99fc-bef89769970a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:08:59.460992 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:59.460968    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bc4b0f8e-790d-412e-99fc-bef89769970a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"bc4b0f8e-790d-412e-99fc-bef89769970a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:08:59.461111 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:59.461037    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bc4b0f8e-790d-412e-99fc-bef89769970a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"bc4b0f8e-790d-412e-99fc-bef89769970a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:08:59.461332 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:59.461308    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bc4b0f8e-790d-412e-99fc-bef89769970a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"bc4b0f8e-790d-412e-99fc-bef89769970a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:08:59.461423 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:59.461359    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bc4b0f8e-790d-412e-99fc-bef89769970a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"bc4b0f8e-790d-412e-99fc-bef89769970a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:08:59.462231 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:59.462207    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bc4b0f8e-790d-412e-99fc-bef89769970a-config-out\") pod \"alertmanager-main-0\" (UID: \"bc4b0f8e-790d-412e-99fc-bef89769970a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:08:59.462787 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:59.462766    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bc4b0f8e-790d-412e-99fc-bef89769970a-config-volume\") pod \"alertmanager-main-0\" (UID: \"bc4b0f8e-790d-412e-99fc-bef89769970a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:08:59.463386 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:59.463364    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/bc4b0f8e-790d-412e-99fc-bef89769970a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"bc4b0f8e-790d-412e-99fc-bef89769970a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:08:59.463711 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:59.463672    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bc4b0f8e-790d-412e-99fc-bef89769970a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"bc4b0f8e-790d-412e-99fc-bef89769970a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:08:59.464260 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:59.464241    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bc4b0f8e-790d-412e-99fc-bef89769970a-web-config\") pod \"alertmanager-main-0\" (UID: \"bc4b0f8e-790d-412e-99fc-bef89769970a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:08:59.482290 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:59.482266    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx8tf\" (UniqueName: \"kubernetes.io/projected/bc4b0f8e-790d-412e-99fc-bef89769970a-kube-api-access-sx8tf\") pod \"alertmanager-main-0\" (UID: \"bc4b0f8e-790d-412e-99fc-bef89769970a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:08:59.552679 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:59.552647    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:08:59.773832 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:59.773736    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-d8vg4" event={"ID":"73013d79-268e-42f8-8bbd-9638f7b29d7a","Type":"ContainerStarted","Data":"d1751895aa77ea52661090a8d26689337ae1380479351c3bda6594fd3c85d57e"}
Jul 28 10:08:59.775173 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:59.775145    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-wt5j9" event={"ID":"8c69d312-24ea-47e7-8f36-1e6a5362010a","Type":"ContainerStarted","Data":"ffd843f16d559795a87d7c232d4832488b507ed5122d63e4c860888508e1dd50"}
Jul 28 10:08:59.962905 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:59.962871    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc4b0f8e-790d-412e-99fc-bef89769970a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"bc4b0f8e-790d-412e-99fc-bef89769970a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:08:59.963917 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:08:59.963893    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc4b0f8e-790d-412e-99fc-bef89769970a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"bc4b0f8e-790d-412e-99fc-bef89769970a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:09:00.183426 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:00.183391    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:09:00.345165 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:00.345141    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 10:09:00.581915 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:09:00.581886    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc4b0f8e_790d_412e_99fc_bef89769970a.slice/crio-2c25a0f8fbc948e91fef6601dd3b5eaa0f2e8d35e8187f1727f4c0816f5b68d6 WatchSource:0}: Error finding container 2c25a0f8fbc948e91fef6601dd3b5eaa0f2e8d35e8187f1727f4c0816f5b68d6: Status 404 returned error can't find the container with id 2c25a0f8fbc948e91fef6601dd3b5eaa0f2e8d35e8187f1727f4c0816f5b68d6
Jul 28 10:09:00.780934 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:00.780902    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-wt5j9" event={"ID":"8c69d312-24ea-47e7-8f36-1e6a5362010a","Type":"ContainerStarted","Data":"0fef62aed361cda3cc4226d1ecda13888222e57f768e91bc9898adc2cbbeacf2"}
Jul 28 10:09:00.781079 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:00.781000    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-wt5j9" event={"ID":"8c69d312-24ea-47e7-8f36-1e6a5362010a","Type":"ContainerStarted","Data":"2f936ed9fcb48ad54f1f780e4535497c8af6c0acc9de992a7e3f8d05b5c230d1"}
Jul 28 10:09:00.782748 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:00.782632    2583 generic.go:356] "Generic (PLEG): container finished" podID="1b50dc93-37d8-40ad-a0c3-f3c36cc72aca" containerID="95927b3ed9db9baba58dd6e614e87b649a28e925cba714ee4cc1edcbc6434784" exitCode=0
Jul 28 10:09:00.782748 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:00.782732    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d8tr2" event={"ID":"1b50dc93-37d8-40ad-a0c3-f3c36cc72aca","Type":"ContainerDied","Data":"95927b3ed9db9baba58dd6e614e87b649a28e925cba714ee4cc1edcbc6434784"}
Jul 28 10:09:00.785841 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:00.785760    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bc4b0f8e-790d-412e-99fc-bef89769970a","Type":"ContainerStarted","Data":"2c25a0f8fbc948e91fef6601dd3b5eaa0f2e8d35e8187f1727f4c0816f5b68d6"}
Jul 28 10:09:00.789412 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:00.789387    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-d8vg4" event={"ID":"73013d79-268e-42f8-8bbd-9638f7b29d7a","Type":"ContainerStarted","Data":"98f0a3dd021e7d92774244489477ad752c1b89cc97ded5d007aaf6f14c5d998f"}
Jul 28 10:09:01.392206 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:01.392150    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:01.538841 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:01.538812    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:01.768225 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:01.768195    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:01.795701 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:01.795673    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d8tr2" event={"ID":"1b50dc93-37d8-40ad-a0c3-f3c36cc72aca","Type":"ContainerStarted","Data":"e1587fe0334897e3e042f6e307162341e17e2b960b42a37214091210744c97ea"}
Jul 28 10:09:01.795805 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:01.795731    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d8tr2" event={"ID":"1b50dc93-37d8-40ad-a0c3-f3c36cc72aca","Type":"ContainerStarted","Data":"ff58e04f7dd0fc22624895969e14edeeecd5b12a6b71fc026a7f9b2c7be1a257"}
Jul 28 10:09:01.797207 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:01.797171    2583 generic.go:356] "Generic (PLEG): container finished" podID="bc4b0f8e-790d-412e-99fc-bef89769970a" containerID="fce5b5f9732a23a9bb236d741da20df05162f5921ef1e326881cc98e8a016f8a" exitCode=0
Jul 28 10:09:01.797294 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:01.797215    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bc4b0f8e-790d-412e-99fc-bef89769970a","Type":"ContainerDied","Data":"fce5b5f9732a23a9bb236d741da20df05162f5921ef1e326881cc98e8a016f8a"}
Jul 28 10:09:01.799642 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:01.799609    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-wt5j9" event={"ID":"8c69d312-24ea-47e7-8f36-1e6a5362010a","Type":"ContainerStarted","Data":"a729726f90423475c3c64d27910d1e76a07f41b0c410db70bd362f284e691458"}
Jul 28 10:09:01.820352 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:01.817391    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-d8vg4" podStartSLOduration=2.764754854 podStartE2EDuration="3.817373511s" totalImagesPullingTime="1.052618657s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:08:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:09:00.883311923 +0000 UTC m=+79.983818490" watchObservedRunningTime="2026-07-28 10:09:01.817373511 +0000 UTC m=+80.917880067"
Jul 28 10:09:01.820352 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:01.817828    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-d8tr2" podStartSLOduration=2.5046551470000002 podStartE2EDuration="3.817816381s" totalImagesPullingTime="1.240200744s" totalInitContainerRuntime="72.96049ms" isStatefulPod=true podCreationTimestamp="2026-07-28 10:08:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:09:01.816972747 +0000 UTC m=+80.917479330" watchObservedRunningTime="2026-07-28 10:09:01.817816381 +0000 UTC m=+80.918322936"
Jul 28 10:09:01.844841 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:01.844801    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-wt5j9" podStartSLOduration=2.543981805 podStartE2EDuration="3.844790546s" totalImagesPullingTime="1.300808741s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:08:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:09:01.843116953 +0000 UTC m=+80.943623528" watchObservedRunningTime="2026-07-28 10:09:01.844790546 +0000 UTC m=+80.945297099"
Jul 28 10:09:01.967564 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:01.967496    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:02.288416 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:02.288315    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:02.389802 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:02.389767    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:02.757543 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:02.757514    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7475fb8885-f76b9"]
Jul 28 10:09:02.760916 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:02.760885    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7475fb8885-f76b9"
Jul 28 10:09:02.766149 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:02.766115    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-jzmqd\""
Jul 28 10:09:02.766149 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:02.766136    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-4rktdn3j932ue\""
Jul 28 10:09:02.766149 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:02.766115    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 28 10:09:02.766371 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:02.766238    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 28 10:09:02.766371 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:02.766136    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 28 10:09:02.766521 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:02.766507    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 28 10:09:02.783120 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:02.783097    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7475fb8885-f76b9"]
Jul 28 10:09:02.887990 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:02.887968    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c57cdfc8-d934-4fed-94bd-d5604c238863-client-ca-bundle\") pod \"metrics-server-7475fb8885-f76b9\" (UID: \"c57cdfc8-d934-4fed-94bd-d5604c238863\") " pod="openshift-monitoring/metrics-server-7475fb8885-f76b9"
Jul 28 10:09:02.888111 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:02.887997    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c57cdfc8-d934-4fed-94bd-d5604c238863-audit-log\") pod \"metrics-server-7475fb8885-f76b9\" (UID: \"c57cdfc8-d934-4fed-94bd-d5604c238863\") " pod="openshift-monitoring/metrics-server-7475fb8885-f76b9"
Jul 28 10:09:02.888174 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:02.888152    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c57cdfc8-d934-4fed-94bd-d5604c238863-secret-metrics-server-tls\") pod \"metrics-server-7475fb8885-f76b9\" (UID: \"c57cdfc8-d934-4fed-94bd-d5604c238863\") " pod="openshift-monitoring/metrics-server-7475fb8885-f76b9"
Jul 28 10:09:02.888243 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:02.888214    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c57cdfc8-d934-4fed-94bd-d5604c238863-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7475fb8885-f76b9\" (UID: \"c57cdfc8-d934-4fed-94bd-d5604c238863\") " pod="openshift-monitoring/metrics-server-7475fb8885-f76b9"
Jul 28 10:09:02.888306 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:02.888287    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c57cdfc8-d934-4fed-94bd-d5604c238863-metrics-server-audit-profiles\") pod \"metrics-server-7475fb8885-f76b9\" (UID: \"c57cdfc8-d934-4fed-94bd-d5604c238863\") " pod="openshift-monitoring/metrics-server-7475fb8885-f76b9"
Jul 28 10:09:02.888616 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:02.888358    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c57cdfc8-d934-4fed-94bd-d5604c238863-secret-metrics-server-client-certs\") pod \"metrics-server-7475fb8885-f76b9\" (UID: \"c57cdfc8-d934-4fed-94bd-d5604c238863\") " pod="openshift-monitoring/metrics-server-7475fb8885-f76b9"
Jul 28 10:09:02.888616 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:02.888483    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlkbr\" (UniqueName: \"kubernetes.io/projected/c57cdfc8-d934-4fed-94bd-d5604c238863-kube-api-access-zlkbr\") pod \"metrics-server-7475fb8885-f76b9\" (UID: \"c57cdfc8-d934-4fed-94bd-d5604c238863\") " pod="openshift-monitoring/metrics-server-7475fb8885-f76b9"
Jul 28 10:09:02.922074 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:02.922045    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:02.989424 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:02.989320    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c57cdfc8-d934-4fed-94bd-d5604c238863-metrics-server-audit-profiles\") pod \"metrics-server-7475fb8885-f76b9\" (UID: \"c57cdfc8-d934-4fed-94bd-d5604c238863\") " pod="openshift-monitoring/metrics-server-7475fb8885-f76b9"
Jul 28 10:09:02.989424 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:02.989362    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c57cdfc8-d934-4fed-94bd-d5604c238863-secret-metrics-server-client-certs\") pod \"metrics-server-7475fb8885-f76b9\" (UID: \"c57cdfc8-d934-4fed-94bd-d5604c238863\") " pod="openshift-monitoring/metrics-server-7475fb8885-f76b9"
Jul 28 10:09:02.989424 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:02.989388    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlkbr\" (UniqueName: \"kubernetes.io/projected/c57cdfc8-d934-4fed-94bd-d5604c238863-kube-api-access-zlkbr\") pod \"metrics-server-7475fb8885-f76b9\" (UID: \"c57cdfc8-d934-4fed-94bd-d5604c238863\") " pod="openshift-monitoring/metrics-server-7475fb8885-f76b9"
Jul 28 10:09:02.989424 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:02.989415    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c57cdfc8-d934-4fed-94bd-d5604c238863-client-ca-bundle\") pod \"metrics-server-7475fb8885-f76b9\" (UID: \"c57cdfc8-d934-4fed-94bd-d5604c238863\") " pod="openshift-monitoring/metrics-server-7475fb8885-f76b9"
Jul 28 10:09:02.989702 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:02.989440    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c57cdfc8-d934-4fed-94bd-d5604c238863-audit-log\") pod \"metrics-server-7475fb8885-f76b9\" (UID: \"c57cdfc8-d934-4fed-94bd-d5604c238863\") " pod="openshift-monitoring/metrics-server-7475fb8885-f76b9"
Jul 28 10:09:02.989702 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:02.989508    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c57cdfc8-d934-4fed-94bd-d5604c238863-secret-metrics-server-tls\") pod \"metrics-server-7475fb8885-f76b9\" (UID: \"c57cdfc8-d934-4fed-94bd-d5604c238863\") " pod="openshift-monitoring/metrics-server-7475fb8885-f76b9"
Jul 28 10:09:02.989702 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:02.989539    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c57cdfc8-d934-4fed-94bd-d5604c238863-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7475fb8885-f76b9\" (UID: \"c57cdfc8-d934-4fed-94bd-d5604c238863\") " pod="openshift-monitoring/metrics-server-7475fb8885-f76b9"
Jul 28 10:09:02.990265 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:02.990238    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c57cdfc8-d934-4fed-94bd-d5604c238863-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7475fb8885-f76b9\" (UID: \"c57cdfc8-d934-4fed-94bd-d5604c238863\") " pod="openshift-monitoring/metrics-server-7475fb8885-f76b9"
Jul 28 10:09:02.990390 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:02.990320    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c57cdfc8-d934-4fed-94bd-d5604c238863-metrics-server-audit-profiles\") pod \"metrics-server-7475fb8885-f76b9\" (UID: \"c57cdfc8-d934-4fed-94bd-d5604c238863\") " pod="openshift-monitoring/metrics-server-7475fb8885-f76b9"
Jul 28 10:09:02.990537 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:02.990521    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c57cdfc8-d934-4fed-94bd-d5604c238863-audit-log\") pod \"metrics-server-7475fb8885-f76b9\" (UID: \"c57cdfc8-d934-4fed-94bd-d5604c238863\") " pod="openshift-monitoring/metrics-server-7475fb8885-f76b9"
Jul 28 10:09:02.992051 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:02.992026    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c57cdfc8-d934-4fed-94bd-d5604c238863-secret-metrics-server-client-certs\") pod \"metrics-server-7475fb8885-f76b9\" (UID: \"c57cdfc8-d934-4fed-94bd-d5604c238863\") " pod="openshift-monitoring/metrics-server-7475fb8885-f76b9"
Jul 28 10:09:02.992156 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:02.992039    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c57cdfc8-d934-4fed-94bd-d5604c238863-secret-metrics-server-tls\") pod \"metrics-server-7475fb8885-f76b9\" (UID: \"c57cdfc8-d934-4fed-94bd-d5604c238863\") " pod="openshift-monitoring/metrics-server-7475fb8885-f76b9"
Jul 28 10:09:02.992156 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:02.992149    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c57cdfc8-d934-4fed-94bd-d5604c238863-client-ca-bundle\") pod \"metrics-server-7475fb8885-f76b9\" (UID: \"c57cdfc8-d934-4fed-94bd-d5604c238863\") " pod="openshift-monitoring/metrics-server-7475fb8885-f76b9"
Jul 28 10:09:03.006630 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:03.006611    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlkbr\" (UniqueName: \"kubernetes.io/projected/c57cdfc8-d934-4fed-94bd-d5604c238863-kube-api-access-zlkbr\") pod \"metrics-server-7475fb8885-f76b9\" (UID: \"c57cdfc8-d934-4fed-94bd-d5604c238863\") " pod="openshift-monitoring/metrics-server-7475fb8885-f76b9"
Jul 28 10:09:03.009675 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:03.009625    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:03.019094 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:03.018812    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-kzfgt"]
Jul 28 10:09:03.022354 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:03.022332    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-kzfgt"
Jul 28 10:09:03.024841 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:03.024815    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 28 10:09:03.025206 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:03.025165    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-j9hf2\""
Jul 28 10:09:03.038873 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:03.038853    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-kzfgt"]
Jul 28 10:09:03.072410 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:03.072386    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7475fb8885-f76b9"
Jul 28 10:09:03.090954 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:03.090930    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/24de5934-2a55-43e8-a482-db62cea970b5-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-kzfgt\" (UID: \"24de5934-2a55-43e8-a482-db62cea970b5\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-kzfgt"
Jul 28 10:09:03.091045 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:03.091029    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rl9v\" (UniqueName: \"kubernetes.io/projected/24de5934-2a55-43e8-a482-db62cea970b5-kube-api-access-7rl9v\") pod \"monitoring-plugin-68b7fc9574-kzfgt\" (UID: \"24de5934-2a55-43e8-a482-db62cea970b5\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-kzfgt"
Jul 28 10:09:03.191514 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:03.191482    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/24de5934-2a55-43e8-a482-db62cea970b5-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-kzfgt\" (UID: \"24de5934-2a55-43e8-a482-db62cea970b5\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-kzfgt"
Jul 28 10:09:03.191670 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:03.191579    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rl9v\" (UniqueName: \"kubernetes.io/projected/24de5934-2a55-43e8-a482-db62cea970b5-kube-api-access-7rl9v\") pod \"monitoring-plugin-68b7fc9574-kzfgt\" (UID: \"24de5934-2a55-43e8-a482-db62cea970b5\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-kzfgt"
Jul 28 10:09:03.194257 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:03.194230    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/24de5934-2a55-43e8-a482-db62cea970b5-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-kzfgt\" (UID: \"24de5934-2a55-43e8-a482-db62cea970b5\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-kzfgt"
Jul 28 10:09:03.200590 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:03.200473    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rl9v\" (UniqueName: \"kubernetes.io/projected/24de5934-2a55-43e8-a482-db62cea970b5-kube-api-access-7rl9v\") pod \"monitoring-plugin-68b7fc9574-kzfgt\" (UID: \"24de5934-2a55-43e8-a482-db62cea970b5\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-kzfgt"
Jul 28 10:09:03.237448 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:03.237370    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:03.331695 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:03.331632    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-kzfgt"
Jul 28 10:09:03.486602 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:03.486573    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-kzfgt"]
Jul 28 10:09:03.488358 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:09:03.488334    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24de5934_2a55_43e8_a482_db62cea970b5.slice/crio-f1a359982a02c5df118c2a242bbe2f003c567942021d61f8e5907880345862a2 WatchSource:0}: Error finding container f1a359982a02c5df118c2a242bbe2f003c567942021d61f8e5907880345862a2: Status 404 returned error can't find the container with id f1a359982a02c5df118c2a242bbe2f003c567942021d61f8e5907880345862a2
Jul 28 10:09:03.505824 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:03.505804    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7475fb8885-f76b9"]
Jul 28 10:09:03.510042 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:09:03.510004    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc57cdfc8_d934_4fed_94bd_d5604c238863.slice/crio-3878f6a38870f3eab6d8866b3bd4b2484b6f2e61dc7515b55512d8c882836047 WatchSource:0}: Error finding container 3878f6a38870f3eab6d8866b3bd4b2484b6f2e61dc7515b55512d8c882836047: Status 404 returned error can't find the container with id 3878f6a38870f3eab6d8866b3bd4b2484b6f2e61dc7515b55512d8c882836047
Jul 28 10:09:03.812066 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:03.812024    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bc4b0f8e-790d-412e-99fc-bef89769970a","Type":"ContainerStarted","Data":"0fe9ef2ae5fe37dab57b05f2a4fed9808bedec4177c361e8d95414826f22d45c"}
Jul 28 10:09:03.812505 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:03.812156    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bc4b0f8e-790d-412e-99fc-bef89769970a","Type":"ContainerStarted","Data":"23caaf1ee5b868febb399b6e65ef1c4227211e1133d17b1c71ff6b94c3ab3091"}
Jul 28 10:09:03.812505 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:03.812230    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bc4b0f8e-790d-412e-99fc-bef89769970a","Type":"ContainerStarted","Data":"c85b39e71f86879ba6850c758a98d972e42ab971a64c4790b5eeeb7a75d12035"}
Jul 28 10:09:03.812505 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:03.812278    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bc4b0f8e-790d-412e-99fc-bef89769970a","Type":"ContainerStarted","Data":"89f3fed3cc78543b278cf3e19314d22d2609fdbff5c0380fc7cc276711ce1d28"}
Jul 28 10:09:03.812505 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:03.812320    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bc4b0f8e-790d-412e-99fc-bef89769970a","Type":"ContainerStarted","Data":"a2819e46d8987c603e5be58f83b80799580bdec7b215d18a6f6908d6d68b6d73"}
Jul 28 10:09:03.813627 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:03.813562    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7475fb8885-f76b9" event={"ID":"c57cdfc8-d934-4fed-94bd-d5604c238863","Type":"ContainerStarted","Data":"3878f6a38870f3eab6d8866b3bd4b2484b6f2e61dc7515b55512d8c882836047"}
Jul 28 10:09:03.814863 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:03.814812    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-kzfgt" event={"ID":"24de5934-2a55-43e8-a482-db62cea970b5","Type":"ContainerStarted","Data":"f1a359982a02c5df118c2a242bbe2f003c567942021d61f8e5907880345862a2"}
Jul 28 10:09:04.737574 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:04.737540    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:05.823717 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:05.823635    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7475fb8885-f76b9" event={"ID":"c57cdfc8-d934-4fed-94bd-d5604c238863","Type":"ContainerStarted","Data":"674580d28f1abafec9f5bc42d9ff645b99f853ff28e75ebe854d4b65116365ef"}
Jul 28 10:09:05.825318 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:05.825292    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-kzfgt" event={"ID":"24de5934-2a55-43e8-a482-db62cea970b5","Type":"ContainerStarted","Data":"f245908487f3d281308ca5c7e7db737fb94ebc405d1f9b65ce7477c4ff6f4dfe"}
Jul 28 10:09:05.825543 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:05.825521    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-kzfgt"
Jul 28 10:09:05.829153 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:05.829131    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bc4b0f8e-790d-412e-99fc-bef89769970a","Type":"ContainerStarted","Data":"c1db9f3c97b5a8d82d00056b2336d52353fa65f328dcc18f00b121526d477df3"}
Jul 28 10:09:05.831015 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:05.830996    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-kzfgt"
Jul 28 10:09:05.877264 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:05.877231    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7475fb8885-f76b9" podStartSLOduration=1.912365984 podStartE2EDuration="3.877218325s" totalImagesPullingTime="1.964852341s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:09:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:09:05.845428671 +0000 UTC m=+84.945935224" watchObservedRunningTime="2026-07-28 10:09:05.877218325 +0000 UTC m=+84.977724879"
Jul 28 10:09:05.877696 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:05.877658    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.483316428 podStartE2EDuration="6.877651316s" totalImagesPullingTime="4.377968441s" totalInitContainerRuntime="16.366447ms" isStatefulPod=false podCreationTimestamp="2026-07-28 10:08:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:09:05.876460698 +0000 UTC m=+84.976967252" watchObservedRunningTime="2026-07-28 10:09:05.877651316 +0000 UTC m=+84.978157869"
Jul 28 10:09:05.895567 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:05.895521    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-kzfgt" podStartSLOduration=1.905754519 podStartE2EDuration="3.895504167s" totalImagesPullingTime="1.989749648s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:09:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:09:05.895337629 +0000 UTC m=+84.995844182" watchObservedRunningTime="2026-07-28 10:09:05.895504167 +0000 UTC m=+84.996010722"
Jul 28 10:09:06.746571 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:06.746542    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:06.844262 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:06.844227    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:06.950297 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:06.950268    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:07.062440 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:07.062374    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:07.391942 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:07.391850    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:07.493750 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:07.493722    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:07.790474 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:07.790437    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:07.888188 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:07.888161    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:08.431698 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:08.431666    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:08.605648 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:08.605599    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:08.868342 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:08.868263    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:09.064346 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:09.064318    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:09.257829 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:09.257802    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:09.369659 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:09.369629    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:09.627290 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:09.627211    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:09.694547 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:09.694521    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:11.534045 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:11.534002    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6b5bff677c-7d2rn" podUID="b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec" containerName="registry" containerID="cri-o://f61bb5ebc20ac69cff774dd2ce13825408fc70e16bdbd659365f424b7fbbbef9" gracePeriod=30
Jul 28 10:09:11.739803 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:11.739780    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-54f7d57974-x8r6k"
Jul 28 10:09:11.757317 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:11.757293    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:11.824581 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:11.824531    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b5bff677c-7d2rn"
Jul 28 10:09:11.835955 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:11.835926    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:11.855705 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:11.855679    2583 generic.go:356] "Generic (PLEG): container finished" podID="b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec" containerID="f61bb5ebc20ac69cff774dd2ce13825408fc70e16bdbd659365f424b7fbbbef9" exitCode=0
Jul 28 10:09:11.855815 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:11.855718    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b5bff677c-7d2rn"
Jul 28 10:09:11.855815 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:11.855754    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b5bff677c-7d2rn" event={"ID":"b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec","Type":"ContainerDied","Data":"f61bb5ebc20ac69cff774dd2ce13825408fc70e16bdbd659365f424b7fbbbef9"}
Jul 28 10:09:11.855815 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:11.855798    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b5bff677c-7d2rn" event={"ID":"b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec","Type":"ContainerDied","Data":"32ec514a1f0845f112f887ed20795c633c98fd309d7025af235c7e08ef691843"}
Jul 28 10:09:11.855966 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:11.855819    2583 scope.go:117] "RemoveContainer" containerID="f61bb5ebc20ac69cff774dd2ce13825408fc70e16bdbd659365f424b7fbbbef9"
Jul 28 10:09:11.864727 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:11.864709    2583 scope.go:117] "RemoveContainer" containerID="f61bb5ebc20ac69cff774dd2ce13825408fc70e16bdbd659365f424b7fbbbef9"
Jul 28 10:09:11.865018 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:09:11.864999    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f61bb5ebc20ac69cff774dd2ce13825408fc70e16bdbd659365f424b7fbbbef9\": container with ID starting with f61bb5ebc20ac69cff774dd2ce13825408fc70e16bdbd659365f424b7fbbbef9 not found: ID does not exist" containerID="f61bb5ebc20ac69cff774dd2ce13825408fc70e16bdbd659365f424b7fbbbef9"
Jul 28 10:09:11.865074 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:11.865027    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f61bb5ebc20ac69cff774dd2ce13825408fc70e16bdbd659365f424b7fbbbef9"} err="failed to get container status \"f61bb5ebc20ac69cff774dd2ce13825408fc70e16bdbd659365f424b7fbbbef9\": rpc error: code = NotFound desc = could not find container \"f61bb5ebc20ac69cff774dd2ce13825408fc70e16bdbd659365f424b7fbbbef9\": container with ID starting with f61bb5ebc20ac69cff774dd2ce13825408fc70e16bdbd659365f424b7fbbbef9 not found: ID does not exist"
Jul 28 10:09:11.912961 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:11.912944    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:11.958009 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:11.957988    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec-registry-certificates\") pod \"b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec\" (UID: \"b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec\") "
Jul 28 10:09:11.958088 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:11.958013    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec-trusted-ca\") pod \"b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec\" (UID: \"b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec\") "
Jul 28 10:09:11.958088 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:11.958052    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec-installation-pull-secrets\") pod \"b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec\" (UID: \"b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec\") "
Jul 28 10:09:11.958159 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:11.958085    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec-registry-tls\") pod \"b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec\" (UID: \"b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec\") "
Jul 28 10:09:11.958159 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:11.958107    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec-bound-sa-token\") pod \"b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec\" (UID: \"b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec\") "
Jul 28 10:09:11.958159 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:11.958140    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-65787\" (UniqueName: \"kubernetes.io/projected/b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec-kube-api-access-65787\") pod \"b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec\" (UID: \"b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec\") "
Jul 28 10:09:11.958287 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:11.958173    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec-ca-trust-extracted\") pod \"b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec\" (UID: \"b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec\") "
Jul 28 10:09:11.958287 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:11.958228    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec-image-registry-private-configuration\") pod \"b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec\" (UID: \"b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec\") "
Jul 28 10:09:11.958484 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:11.958440    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec" (UID: "b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:09:11.958484 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:11.958457    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec" (UID: "b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:09:11.960580 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:11.960555    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec" (UID: "b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:09:11.960681 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:11.960650    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec" (UID: "b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:09:11.960781 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:11.960761    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec" (UID: "b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:09:11.960833 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:11.960784    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec" (UID: "b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:09:11.961053 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:11.961034    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec-kube-api-access-65787" (OuterVolumeSpecName: "kube-api-access-65787") pod "b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec" (UID: "b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec"). InnerVolumeSpecName "kube-api-access-65787". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:09:11.966948 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:11.966923    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec" (UID: "b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:09:12.059069 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:12.059049    2583 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec-registry-tls\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:09:12.059069 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:12.059068    2583 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec-bound-sa-token\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:09:12.059196 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:12.059077    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65787\" (UniqueName: \"kubernetes.io/projected/b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec-kube-api-access-65787\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:09:12.059196 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:12.059086    2583 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec-ca-trust-extracted\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:09:12.059196 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:12.059095    2583 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec-image-registry-private-configuration\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:09:12.059196 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:12.059104    2583 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec-registry-certificates\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:09:12.059196 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:12.059113    2583 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec-trusted-ca\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:09:12.059196 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:12.059121    2583 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec-installation-pull-secrets\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:09:12.149995 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:12.149973    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:12.179008 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:12.178987    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6b5bff677c-7d2rn"]
Jul 28 10:09:12.183145 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:12.183122    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6b5bff677c-7d2rn"]
Jul 28 10:09:12.228051 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:12.228027    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:13.437304 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:13.437273    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec" path="/var/lib/kubelet/pods/b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec/volumes"
Jul 28 10:09:17.277111 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:17.277073    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:17.348052 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:17.348021    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:19.212621 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:19.212588    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:21.150510 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:21.150476    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:21.254228 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:21.254172    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:22.226697 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:22.226655    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:22.331080 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:22.331049    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:23.073293 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:23.073263    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7475fb8885-f76b9"
Jul 28 10:09:23.073458 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:23.073317    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7475fb8885-f76b9"
Jul 28 10:09:25.754907 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:25.754878    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-7dfzm"
Jul 28 10:09:26.620928 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:26.620896    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:26.721463 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:26.721432    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:32.252170 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:32.252137    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:32.348697 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:32.348668    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:36.823991 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:36.823952    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:36.882494 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:36.882435    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:38.220394 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:38.220360    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:39.170828 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:39.170802    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:09:43.077492 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:43.077463    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7475fb8885-f76b9"
Jul 28 10:09:43.081135 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:09:43.081110    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7475fb8885-f76b9"
Jul 28 10:10:02.269860 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:02.269830    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:10:02.437435 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:02.437407    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:10:17.718872 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:17.718819    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:10:18.082960 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:18.082860    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:10:18.763950 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:18.763906    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 10:10:18.764538 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:18.764503    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="bc4b0f8e-790d-412e-99fc-bef89769970a" containerName="alertmanager" containerID="cri-o://a2819e46d8987c603e5be58f83b80799580bdec7b215d18a6f6908d6d68b6d73" gracePeriod=120
Jul 28 10:10:18.764910 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:18.764880    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="bc4b0f8e-790d-412e-99fc-bef89769970a" containerName="kube-rbac-proxy-web" containerID="cri-o://c85b39e71f86879ba6850c758a98d972e42ab971a64c4790b5eeeb7a75d12035" gracePeriod=120
Jul 28 10:10:18.765049 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:18.765030    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="bc4b0f8e-790d-412e-99fc-bef89769970a" containerName="config-reloader" containerID="cri-o://89f3fed3cc78543b278cf3e19314d22d2609fdbff5c0380fc7cc276711ce1d28" gracePeriod=120
Jul 28 10:10:18.765485 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:18.765206    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="bc4b0f8e-790d-412e-99fc-bef89769970a" containerName="prom-label-proxy" containerID="cri-o://c1db9f3c97b5a8d82d00056b2336d52353fa65f328dcc18f00b121526d477df3" gracePeriod=120
Jul 28 10:10:18.765485 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:18.765358    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="bc4b0f8e-790d-412e-99fc-bef89769970a" containerName="kube-rbac-proxy-metric" containerID="cri-o://0fe9ef2ae5fe37dab57b05f2a4fed9808bedec4177c361e8d95414826f22d45c" gracePeriod=120
Jul 28 10:10:18.765945 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:18.765881    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="bc4b0f8e-790d-412e-99fc-bef89769970a" containerName="kube-rbac-proxy" containerID="cri-o://23caaf1ee5b868febb399b6e65ef1c4227211e1133d17b1c71ff6b94c3ab3091" gracePeriod=120
Jul 28 10:10:19.101634 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:19.101539    2583 generic.go:356] "Generic (PLEG): container finished" podID="bc4b0f8e-790d-412e-99fc-bef89769970a" containerID="c1db9f3c97b5a8d82d00056b2336d52353fa65f328dcc18f00b121526d477df3" exitCode=0
Jul 28 10:10:19.101634 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:19.101562    2583 generic.go:356] "Generic (PLEG): container finished" podID="bc4b0f8e-790d-412e-99fc-bef89769970a" containerID="23caaf1ee5b868febb399b6e65ef1c4227211e1133d17b1c71ff6b94c3ab3091" exitCode=0
Jul 28 10:10:19.101634 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:19.101570    2583 generic.go:356] "Generic (PLEG): container finished" podID="bc4b0f8e-790d-412e-99fc-bef89769970a" containerID="89f3fed3cc78543b278cf3e19314d22d2609fdbff5c0380fc7cc276711ce1d28" exitCode=0
Jul 28 10:10:19.101634 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:19.101576    2583 generic.go:356] "Generic (PLEG): container finished" podID="bc4b0f8e-790d-412e-99fc-bef89769970a" containerID="a2819e46d8987c603e5be58f83b80799580bdec7b215d18a6f6908d6d68b6d73" exitCode=0
Jul 28 10:10:19.101634 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:19.101610    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bc4b0f8e-790d-412e-99fc-bef89769970a","Type":"ContainerDied","Data":"c1db9f3c97b5a8d82d00056b2336d52353fa65f328dcc18f00b121526d477df3"}
Jul 28 10:10:19.101930 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:19.101688    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bc4b0f8e-790d-412e-99fc-bef89769970a","Type":"ContainerDied","Data":"23caaf1ee5b868febb399b6e65ef1c4227211e1133d17b1c71ff6b94c3ab3091"}
Jul 28 10:10:19.101930 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:19.101727    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bc4b0f8e-790d-412e-99fc-bef89769970a","Type":"ContainerDied","Data":"89f3fed3cc78543b278cf3e19314d22d2609fdbff5c0380fc7cc276711ce1d28"}
Jul 28 10:10:19.101930 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:19.101764    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bc4b0f8e-790d-412e-99fc-bef89769970a","Type":"ContainerDied","Data":"a2819e46d8987c603e5be58f83b80799580bdec7b215d18a6f6908d6d68b6d73"}
Jul 28 10:10:20.019290 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.019266    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:10:20.059716 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.059634    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bc4b0f8e-790d-412e-99fc-bef89769970a-config-volume\") pod \"bc4b0f8e-790d-412e-99fc-bef89769970a\" (UID: \"bc4b0f8e-790d-412e-99fc-bef89769970a\") "
Jul 28 10:10:20.059716 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.059675    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bc4b0f8e-790d-412e-99fc-bef89769970a-secret-alertmanager-main-tls\") pod \"bc4b0f8e-790d-412e-99fc-bef89769970a\" (UID: \"bc4b0f8e-790d-412e-99fc-bef89769970a\") "
Jul 28 10:10:20.059716 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.059696    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bc4b0f8e-790d-412e-99fc-bef89769970a-config-out\") pod \"bc4b0f8e-790d-412e-99fc-bef89769970a\" (UID: \"bc4b0f8e-790d-412e-99fc-bef89769970a\") "
Jul 28 10:10:20.059968 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.059731    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc4b0f8e-790d-412e-99fc-bef89769970a-alertmanager-trusted-ca-bundle\") pod \"bc4b0f8e-790d-412e-99fc-bef89769970a\" (UID: \"bc4b0f8e-790d-412e-99fc-bef89769970a\") "
Jul 28 10:10:20.060593 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.060215    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bc4b0f8e-790d-412e-99fc-bef89769970a-secret-alertmanager-kube-rbac-proxy-web\") pod \"bc4b0f8e-790d-412e-99fc-bef89769970a\" (UID: \"bc4b0f8e-790d-412e-99fc-bef89769970a\") "
Jul 28 10:10:20.060593 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.060278    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bc4b0f8e-790d-412e-99fc-bef89769970a-tls-assets\") pod \"bc4b0f8e-790d-412e-99fc-bef89769970a\" (UID: \"bc4b0f8e-790d-412e-99fc-bef89769970a\") "
Jul 28 10:10:20.060593 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.060307    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bc4b0f8e-790d-412e-99fc-bef89769970a-secret-alertmanager-kube-rbac-proxy\") pod \"bc4b0f8e-790d-412e-99fc-bef89769970a\" (UID: \"bc4b0f8e-790d-412e-99fc-bef89769970a\") "
Jul 28 10:10:20.060593 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.060334    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/bc4b0f8e-790d-412e-99fc-bef89769970a-alertmanager-main-db\") pod \"bc4b0f8e-790d-412e-99fc-bef89769970a\" (UID: \"bc4b0f8e-790d-412e-99fc-bef89769970a\") "
Jul 28 10:10:20.060593 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.060360    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bc4b0f8e-790d-412e-99fc-bef89769970a-web-config\") pod \"bc4b0f8e-790d-412e-99fc-bef89769970a\" (UID: \"bc4b0f8e-790d-412e-99fc-bef89769970a\") "
Jul 28 10:10:20.060593 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.060418    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/bc4b0f8e-790d-412e-99fc-bef89769970a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"bc4b0f8e-790d-412e-99fc-bef89769970a\" (UID: \"bc4b0f8e-790d-412e-99fc-bef89769970a\") "
Jul 28 10:10:20.060593 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.060444    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sx8tf\" (UniqueName: \"kubernetes.io/projected/bc4b0f8e-790d-412e-99fc-bef89769970a-kube-api-access-sx8tf\") pod \"bc4b0f8e-790d-412e-99fc-bef89769970a\" (UID: \"bc4b0f8e-790d-412e-99fc-bef89769970a\") "
Jul 28 10:10:20.060593 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.060467    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bc4b0f8e-790d-412e-99fc-bef89769970a-cluster-tls-config\") pod \"bc4b0f8e-790d-412e-99fc-bef89769970a\" (UID: \"bc4b0f8e-790d-412e-99fc-bef89769970a\") "
Jul 28 10:10:20.060593 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.060512    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bc4b0f8e-790d-412e-99fc-bef89769970a-metrics-client-ca\") pod \"bc4b0f8e-790d-412e-99fc-bef89769970a\" (UID: \"bc4b0f8e-790d-412e-99fc-bef89769970a\") "
Jul 28 10:10:20.061823 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.060262    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc4b0f8e-790d-412e-99fc-bef89769970a-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "bc4b0f8e-790d-412e-99fc-bef89769970a" (UID: "bc4b0f8e-790d-412e-99fc-bef89769970a"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:10:20.061823 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.061073    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc4b0f8e-790d-412e-99fc-bef89769970a-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "bc4b0f8e-790d-412e-99fc-bef89769970a" (UID: "bc4b0f8e-790d-412e-99fc-bef89769970a"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:10:20.061993 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.061940    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc4b0f8e-790d-412e-99fc-bef89769970a-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "bc4b0f8e-790d-412e-99fc-bef89769970a" (UID: "bc4b0f8e-790d-412e-99fc-bef89769970a"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:10:20.062552 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.062517    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc4b0f8e-790d-412e-99fc-bef89769970a-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "bc4b0f8e-790d-412e-99fc-bef89769970a" (UID: "bc4b0f8e-790d-412e-99fc-bef89769970a"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:10:20.062752 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.062717    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc4b0f8e-790d-412e-99fc-bef89769970a-config-volume" (OuterVolumeSpecName: "config-volume") pod "bc4b0f8e-790d-412e-99fc-bef89769970a" (UID: "bc4b0f8e-790d-412e-99fc-bef89769970a"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:10:20.063331 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.063292    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc4b0f8e-790d-412e-99fc-bef89769970a-config-out" (OuterVolumeSpecName: "config-out") pod "bc4b0f8e-790d-412e-99fc-bef89769970a" (UID: "bc4b0f8e-790d-412e-99fc-bef89769970a"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:10:20.063896 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.063835    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc4b0f8e-790d-412e-99fc-bef89769970a-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "bc4b0f8e-790d-412e-99fc-bef89769970a" (UID: "bc4b0f8e-790d-412e-99fc-bef89769970a"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:10:20.065226 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.065121    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc4b0f8e-790d-412e-99fc-bef89769970a-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "bc4b0f8e-790d-412e-99fc-bef89769970a" (UID: "bc4b0f8e-790d-412e-99fc-bef89769970a"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:10:20.065226 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.065170    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc4b0f8e-790d-412e-99fc-bef89769970a-kube-api-access-sx8tf" (OuterVolumeSpecName: "kube-api-access-sx8tf") pod "bc4b0f8e-790d-412e-99fc-bef89769970a" (UID: "bc4b0f8e-790d-412e-99fc-bef89769970a"). InnerVolumeSpecName "kube-api-access-sx8tf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:10:20.066007 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.065976    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc4b0f8e-790d-412e-99fc-bef89769970a-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "bc4b0f8e-790d-412e-99fc-bef89769970a" (UID: "bc4b0f8e-790d-412e-99fc-bef89769970a"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:10:20.066287 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.066264    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc4b0f8e-790d-412e-99fc-bef89769970a-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "bc4b0f8e-790d-412e-99fc-bef89769970a" (UID: "bc4b0f8e-790d-412e-99fc-bef89769970a"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:10:20.069620 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.069589    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc4b0f8e-790d-412e-99fc-bef89769970a-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "bc4b0f8e-790d-412e-99fc-bef89769970a" (UID: "bc4b0f8e-790d-412e-99fc-bef89769970a"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:10:20.076215 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.076167    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc4b0f8e-790d-412e-99fc-bef89769970a-web-config" (OuterVolumeSpecName: "web-config") pod "bc4b0f8e-790d-412e-99fc-bef89769970a" (UID: "bc4b0f8e-790d-412e-99fc-bef89769970a"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:10:20.108432 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.108398    2583 generic.go:356] "Generic (PLEG): container finished" podID="bc4b0f8e-790d-412e-99fc-bef89769970a" containerID="0fe9ef2ae5fe37dab57b05f2a4fed9808bedec4177c361e8d95414826f22d45c" exitCode=0
Jul 28 10:10:20.108432 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.108426    2583 generic.go:356] "Generic (PLEG): container finished" podID="bc4b0f8e-790d-412e-99fc-bef89769970a" containerID="c85b39e71f86879ba6850c758a98d972e42ab971a64c4790b5eeeb7a75d12035" exitCode=0
Jul 28 10:10:20.108619 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.108455    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bc4b0f8e-790d-412e-99fc-bef89769970a","Type":"ContainerDied","Data":"0fe9ef2ae5fe37dab57b05f2a4fed9808bedec4177c361e8d95414826f22d45c"}
Jul 28 10:10:20.108619 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.108483    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:10:20.108619 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.108531    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bc4b0f8e-790d-412e-99fc-bef89769970a","Type":"ContainerDied","Data":"c85b39e71f86879ba6850c758a98d972e42ab971a64c4790b5eeeb7a75d12035"}
Jul 28 10:10:20.108619 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.108571    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bc4b0f8e-790d-412e-99fc-bef89769970a","Type":"ContainerDied","Data":"2c25a0f8fbc948e91fef6601dd3b5eaa0f2e8d35e8187f1727f4c0816f5b68d6"}
Jul 28 10:10:20.108619 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.108608    2583 scope.go:117] "RemoveContainer" containerID="c1db9f3c97b5a8d82d00056b2336d52353fa65f328dcc18f00b121526d477df3"
Jul 28 10:10:20.116297 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.116277    2583 scope.go:117] "RemoveContainer" containerID="0fe9ef2ae5fe37dab57b05f2a4fed9808bedec4177c361e8d95414826f22d45c"
Jul 28 10:10:20.123320 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.123287    2583 scope.go:117] "RemoveContainer" containerID="23caaf1ee5b868febb399b6e65ef1c4227211e1133d17b1c71ff6b94c3ab3091"
Jul 28 10:10:20.129768 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.129750    2583 scope.go:117] "RemoveContainer" containerID="c85b39e71f86879ba6850c758a98d972e42ab971a64c4790b5eeeb7a75d12035"
Jul 28 10:10:20.134214 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.134159    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 10:10:20.136687 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.136659    2583 scope.go:117] "RemoveContainer" containerID="89f3fed3cc78543b278cf3e19314d22d2609fdbff5c0380fc7cc276711ce1d28"
Jul 28 10:10:20.140542 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.140523    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 10:10:20.143965 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.143949    2583 scope.go:117] "RemoveContainer" containerID="a2819e46d8987c603e5be58f83b80799580bdec7b215d18a6f6908d6d68b6d73"
Jul 28 10:10:20.150657 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.150641    2583 scope.go:117] "RemoveContainer" containerID="fce5b5f9732a23a9bb236d741da20df05162f5921ef1e326881cc98e8a016f8a"
Jul 28 10:10:20.157162 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.157145    2583 scope.go:117] "RemoveContainer" containerID="c1db9f3c97b5a8d82d00056b2336d52353fa65f328dcc18f00b121526d477df3"
Jul 28 10:10:20.157451 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:10:20.157433    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1db9f3c97b5a8d82d00056b2336d52353fa65f328dcc18f00b121526d477df3\": container with ID starting with c1db9f3c97b5a8d82d00056b2336d52353fa65f328dcc18f00b121526d477df3 not found: ID does not exist" containerID="c1db9f3c97b5a8d82d00056b2336d52353fa65f328dcc18f00b121526d477df3"
Jul 28 10:10:20.157514 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.157460    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1db9f3c97b5a8d82d00056b2336d52353fa65f328dcc18f00b121526d477df3"} err="failed to get container status \"c1db9f3c97b5a8d82d00056b2336d52353fa65f328dcc18f00b121526d477df3\": rpc error: code = NotFound desc = could not find container \"c1db9f3c97b5a8d82d00056b2336d52353fa65f328dcc18f00b121526d477df3\": container with ID starting with c1db9f3c97b5a8d82d00056b2336d52353fa65f328dcc18f00b121526d477df3 not found: ID does not exist"
Jul 28 10:10:20.157514 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.157480    2583 scope.go:117] "RemoveContainer" containerID="0fe9ef2ae5fe37dab57b05f2a4fed9808bedec4177c361e8d95414826f22d45c"
Jul 28 10:10:20.157718 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:10:20.157703    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fe9ef2ae5fe37dab57b05f2a4fed9808bedec4177c361e8d95414826f22d45c\": container with ID starting with 0fe9ef2ae5fe37dab57b05f2a4fed9808bedec4177c361e8d95414826f22d45c not found: ID does not exist" containerID="0fe9ef2ae5fe37dab57b05f2a4fed9808bedec4177c361e8d95414826f22d45c"
Jul 28 10:10:20.157754 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.157730    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fe9ef2ae5fe37dab57b05f2a4fed9808bedec4177c361e8d95414826f22d45c"} err="failed to get container status \"0fe9ef2ae5fe37dab57b05f2a4fed9808bedec4177c361e8d95414826f22d45c\": rpc error: code = NotFound desc = could not find container \"0fe9ef2ae5fe37dab57b05f2a4fed9808bedec4177c361e8d95414826f22d45c\": container with ID starting with 0fe9ef2ae5fe37dab57b05f2a4fed9808bedec4177c361e8d95414826f22d45c not found: ID does not exist"
Jul 28 10:10:20.157754 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.157746    2583 scope.go:117] "RemoveContainer" containerID="23caaf1ee5b868febb399b6e65ef1c4227211e1133d17b1c71ff6b94c3ab3091"
Jul 28 10:10:20.157970 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:10:20.157951    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23caaf1ee5b868febb399b6e65ef1c4227211e1133d17b1c71ff6b94c3ab3091\": container with ID starting with 23caaf1ee5b868febb399b6e65ef1c4227211e1133d17b1c71ff6b94c3ab3091 not found: ID does not exist" containerID="23caaf1ee5b868febb399b6e65ef1c4227211e1133d17b1c71ff6b94c3ab3091"
Jul 28 10:10:20.158009 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.157975    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23caaf1ee5b868febb399b6e65ef1c4227211e1133d17b1c71ff6b94c3ab3091"} err="failed to get container status \"23caaf1ee5b868febb399b6e65ef1c4227211e1133d17b1c71ff6b94c3ab3091\": rpc error: code = NotFound desc = could not find container \"23caaf1ee5b868febb399b6e65ef1c4227211e1133d17b1c71ff6b94c3ab3091\": container with ID starting with 23caaf1ee5b868febb399b6e65ef1c4227211e1133d17b1c71ff6b94c3ab3091 not found: ID does not exist"
Jul 28 10:10:20.158009 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.157991    2583 scope.go:117] "RemoveContainer" containerID="c85b39e71f86879ba6850c758a98d972e42ab971a64c4790b5eeeb7a75d12035"
Jul 28 10:10:20.158261 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:10:20.158246    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c85b39e71f86879ba6850c758a98d972e42ab971a64c4790b5eeeb7a75d12035\": container with ID starting with c85b39e71f86879ba6850c758a98d972e42ab971a64c4790b5eeeb7a75d12035 not found: ID does not exist" containerID="c85b39e71f86879ba6850c758a98d972e42ab971a64c4790b5eeeb7a75d12035"
Jul 28 10:10:20.158301 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.158265    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c85b39e71f86879ba6850c758a98d972e42ab971a64c4790b5eeeb7a75d12035"} err="failed to get container status \"c85b39e71f86879ba6850c758a98d972e42ab971a64c4790b5eeeb7a75d12035\": rpc error: code = NotFound desc = could not find container \"c85b39e71f86879ba6850c758a98d972e42ab971a64c4790b5eeeb7a75d12035\": container with ID starting with c85b39e71f86879ba6850c758a98d972e42ab971a64c4790b5eeeb7a75d12035 not found: ID does not exist"
Jul 28 10:10:20.158301 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.158281    2583 scope.go:117] "RemoveContainer" containerID="89f3fed3cc78543b278cf3e19314d22d2609fdbff5c0380fc7cc276711ce1d28"
Jul 28 10:10:20.158493 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:10:20.158477    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89f3fed3cc78543b278cf3e19314d22d2609fdbff5c0380fc7cc276711ce1d28\": container with ID starting with 89f3fed3cc78543b278cf3e19314d22d2609fdbff5c0380fc7cc276711ce1d28 not found: ID does not exist" containerID="89f3fed3cc78543b278cf3e19314d22d2609fdbff5c0380fc7cc276711ce1d28"
Jul 28 10:10:20.158532 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.158496    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89f3fed3cc78543b278cf3e19314d22d2609fdbff5c0380fc7cc276711ce1d28"} err="failed to get container status \"89f3fed3cc78543b278cf3e19314d22d2609fdbff5c0380fc7cc276711ce1d28\": rpc error: code = NotFound desc = could not find container \"89f3fed3cc78543b278cf3e19314d22d2609fdbff5c0380fc7cc276711ce1d28\": container with ID starting with 89f3fed3cc78543b278cf3e19314d22d2609fdbff5c0380fc7cc276711ce1d28 not found: ID does not exist"
Jul 28 10:10:20.158532 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.158508    2583 scope.go:117] "RemoveContainer" containerID="a2819e46d8987c603e5be58f83b80799580bdec7b215d18a6f6908d6d68b6d73"
Jul 28 10:10:20.158688 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:10:20.158674    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2819e46d8987c603e5be58f83b80799580bdec7b215d18a6f6908d6d68b6d73\": container with ID starting with a2819e46d8987c603e5be58f83b80799580bdec7b215d18a6f6908d6d68b6d73 not found: ID does not exist" containerID="a2819e46d8987c603e5be58f83b80799580bdec7b215d18a6f6908d6d68b6d73"
Jul 28 10:10:20.158722 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.158693    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2819e46d8987c603e5be58f83b80799580bdec7b215d18a6f6908d6d68b6d73"} err="failed to get container status \"a2819e46d8987c603e5be58f83b80799580bdec7b215d18a6f6908d6d68b6d73\": rpc error: code = NotFound desc = could not find container \"a2819e46d8987c603e5be58f83b80799580bdec7b215d18a6f6908d6d68b6d73\": container with ID starting with a2819e46d8987c603e5be58f83b80799580bdec7b215d18a6f6908d6d68b6d73 not found: ID does not exist"
Jul 28 10:10:20.158722 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.158707    2583 scope.go:117] "RemoveContainer" containerID="fce5b5f9732a23a9bb236d741da20df05162f5921ef1e326881cc98e8a016f8a"
Jul 28 10:10:20.158909 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:10:20.158894    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fce5b5f9732a23a9bb236d741da20df05162f5921ef1e326881cc98e8a016f8a\": container with ID starting with fce5b5f9732a23a9bb236d741da20df05162f5921ef1e326881cc98e8a016f8a not found: ID does not exist" containerID="fce5b5f9732a23a9bb236d741da20df05162f5921ef1e326881cc98e8a016f8a"
Jul 28 10:10:20.158942 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.158912    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fce5b5f9732a23a9bb236d741da20df05162f5921ef1e326881cc98e8a016f8a"} err="failed to get container status \"fce5b5f9732a23a9bb236d741da20df05162f5921ef1e326881cc98e8a016f8a\": rpc error: code = NotFound desc = could not find container \"fce5b5f9732a23a9bb236d741da20df05162f5921ef1e326881cc98e8a016f8a\": container with ID starting with fce5b5f9732a23a9bb236d741da20df05162f5921ef1e326881cc98e8a016f8a not found: ID does not exist"
Jul 28 10:10:20.158942 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.158927    2583 scope.go:117] "RemoveContainer" containerID="c1db9f3c97b5a8d82d00056b2336d52353fa65f328dcc18f00b121526d477df3"
Jul 28 10:10:20.159121 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.159107    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1db9f3c97b5a8d82d00056b2336d52353fa65f328dcc18f00b121526d477df3"} err="failed to get container status \"c1db9f3c97b5a8d82d00056b2336d52353fa65f328dcc18f00b121526d477df3\": rpc error: code = NotFound desc = could not find container \"c1db9f3c97b5a8d82d00056b2336d52353fa65f328dcc18f00b121526d477df3\": container with ID starting with c1db9f3c97b5a8d82d00056b2336d52353fa65f328dcc18f00b121526d477df3 not found: ID does not exist"
Jul 28 10:10:20.159163 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.159122    2583 scope.go:117] "RemoveContainer" containerID="0fe9ef2ae5fe37dab57b05f2a4fed9808bedec4177c361e8d95414826f22d45c"
Jul 28 10:10:20.159339 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.159321    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fe9ef2ae5fe37dab57b05f2a4fed9808bedec4177c361e8d95414826f22d45c"} err="failed to get container status \"0fe9ef2ae5fe37dab57b05f2a4fed9808bedec4177c361e8d95414826f22d45c\": rpc error: code = NotFound desc = could not find container \"0fe9ef2ae5fe37dab57b05f2a4fed9808bedec4177c361e8d95414826f22d45c\": container with ID starting with 0fe9ef2ae5fe37dab57b05f2a4fed9808bedec4177c361e8d95414826f22d45c not found: ID does not exist"
Jul 28 10:10:20.159412 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.159342    2583 scope.go:117] "RemoveContainer" containerID="23caaf1ee5b868febb399b6e65ef1c4227211e1133d17b1c71ff6b94c3ab3091"
Jul 28 10:10:20.159544 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.159528    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23caaf1ee5b868febb399b6e65ef1c4227211e1133d17b1c71ff6b94c3ab3091"} err="failed to get container status \"23caaf1ee5b868febb399b6e65ef1c4227211e1133d17b1c71ff6b94c3ab3091\": rpc error: code = NotFound desc = could not find container \"23caaf1ee5b868febb399b6e65ef1c4227211e1133d17b1c71ff6b94c3ab3091\": container with ID starting with 23caaf1ee5b868febb399b6e65ef1c4227211e1133d17b1c71ff6b94c3ab3091 not found: ID does not exist"
Jul 28 10:10:20.159592 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.159544    2583 scope.go:117] "RemoveContainer" containerID="c85b39e71f86879ba6850c758a98d972e42ab971a64c4790b5eeeb7a75d12035"
Jul 28 10:10:20.159745 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.159723    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c85b39e71f86879ba6850c758a98d972e42ab971a64c4790b5eeeb7a75d12035"} err="failed to get container status \"c85b39e71f86879ba6850c758a98d972e42ab971a64c4790b5eeeb7a75d12035\": rpc error: code = NotFound desc = could not find container \"c85b39e71f86879ba6850c758a98d972e42ab971a64c4790b5eeeb7a75d12035\": container with ID starting with c85b39e71f86879ba6850c758a98d972e42ab971a64c4790b5eeeb7a75d12035 not found: ID does not exist"
Jul 28 10:10:20.159796 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.159746    2583 scope.go:117] "RemoveContainer" containerID="89f3fed3cc78543b278cf3e19314d22d2609fdbff5c0380fc7cc276711ce1d28"
Jul 28 10:10:20.159941 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.159924    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89f3fed3cc78543b278cf3e19314d22d2609fdbff5c0380fc7cc276711ce1d28"} err="failed to get container status \"89f3fed3cc78543b278cf3e19314d22d2609fdbff5c0380fc7cc276711ce1d28\": rpc error: code = NotFound desc = could not find container \"89f3fed3cc78543b278cf3e19314d22d2609fdbff5c0380fc7cc276711ce1d28\": container with ID starting with 89f3fed3cc78543b278cf3e19314d22d2609fdbff5c0380fc7cc276711ce1d28 not found: ID does not exist"
Jul 28 10:10:20.160008 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.159943    2583 scope.go:117] "RemoveContainer" containerID="a2819e46d8987c603e5be58f83b80799580bdec7b215d18a6f6908d6d68b6d73"
Jul 28 10:10:20.160159 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.160143    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2819e46d8987c603e5be58f83b80799580bdec7b215d18a6f6908d6d68b6d73"} err="failed to get container status \"a2819e46d8987c603e5be58f83b80799580bdec7b215d18a6f6908d6d68b6d73\": rpc error: code = NotFound desc = could not find container \"a2819e46d8987c603e5be58f83b80799580bdec7b215d18a6f6908d6d68b6d73\": container with ID starting with a2819e46d8987c603e5be58f83b80799580bdec7b215d18a6f6908d6d68b6d73 not found: ID does not exist"
Jul 28 10:10:20.160223 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.160159    2583 scope.go:117] "RemoveContainer" containerID="fce5b5f9732a23a9bb236d741da20df05162f5921ef1e326881cc98e8a016f8a"
Jul 28 10:10:20.160393 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.160374    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fce5b5f9732a23a9bb236d741da20df05162f5921ef1e326881cc98e8a016f8a"} err="failed to get container status \"fce5b5f9732a23a9bb236d741da20df05162f5921ef1e326881cc98e8a016f8a\": rpc error: code = NotFound desc = could not find container \"fce5b5f9732a23a9bb236d741da20df05162f5921ef1e326881cc98e8a016f8a\": container with ID starting with fce5b5f9732a23a9bb236d741da20df05162f5921ef1e326881cc98e8a016f8a not found: ID does not exist"
Jul 28 10:10:20.161732 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.161714    2583 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bc4b0f8e-790d-412e-99fc-bef89769970a-config-volume\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:10:20.161732 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.161733    2583 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bc4b0f8e-790d-412e-99fc-bef89769970a-secret-alertmanager-main-tls\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:10:20.161837 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.161743    2583 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bc4b0f8e-790d-412e-99fc-bef89769970a-config-out\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:10:20.161837 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.161757    2583 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc4b0f8e-790d-412e-99fc-bef89769970a-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:10:20.161837 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.161770    2583 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bc4b0f8e-790d-412e-99fc-bef89769970a-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:10:20.161837 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.161779    2583 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bc4b0f8e-790d-412e-99fc-bef89769970a-tls-assets\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:10:20.161837 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.161788    2583 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bc4b0f8e-790d-412e-99fc-bef89769970a-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:10:20.161837 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.161796    2583 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/bc4b0f8e-790d-412e-99fc-bef89769970a-alertmanager-main-db\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:10:20.161837 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.161804    2583 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bc4b0f8e-790d-412e-99fc-bef89769970a-web-config\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:10:20.161837 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.161812    2583 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/bc4b0f8e-790d-412e-99fc-bef89769970a-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:10:20.161837 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.161820    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sx8tf\" (UniqueName: \"kubernetes.io/projected/bc4b0f8e-790d-412e-99fc-bef89769970a-kube-api-access-sx8tf\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:10:20.161837 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.161830    2583 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bc4b0f8e-790d-412e-99fc-bef89769970a-cluster-tls-config\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:10:20.161837 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.161838    2583 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bc4b0f8e-790d-412e-99fc-bef89769970a-metrics-client-ca\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:10:20.171498 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.171472    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 10:10:20.171721 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.171710    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc4b0f8e-790d-412e-99fc-bef89769970a" containerName="kube-rbac-proxy-metric"
Jul 28 10:10:20.171763 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.171723    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc4b0f8e-790d-412e-99fc-bef89769970a" containerName="kube-rbac-proxy-metric"
Jul 28 10:10:20.171763 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.171734    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc4b0f8e-790d-412e-99fc-bef89769970a" containerName="alertmanager"
Jul 28 10:10:20.171763 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.171740    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc4b0f8e-790d-412e-99fc-bef89769970a" containerName="alertmanager"
Jul 28 10:10:20.171763 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.171750    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc4b0f8e-790d-412e-99fc-bef89769970a" containerName="kube-rbac-proxy"
Jul 28 10:10:20.171763 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.171756    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc4b0f8e-790d-412e-99fc-bef89769970a" containerName="kube-rbac-proxy"
Jul 28 10:10:20.171763 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.171764    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc4b0f8e-790d-412e-99fc-bef89769970a" containerName="prom-label-proxy"
Jul 28 10:10:20.171934 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.171770    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc4b0f8e-790d-412e-99fc-bef89769970a" containerName="prom-label-proxy"
Jul 28 10:10:20.171934 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.171779    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec" containerName="registry"
Jul 28 10:10:20.171934 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.171784    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec" containerName="registry"
Jul 28 10:10:20.171934 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.171792    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc4b0f8e-790d-412e-99fc-bef89769970a" containerName="init-config-reloader"
Jul 28 10:10:20.171934 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.171797    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc4b0f8e-790d-412e-99fc-bef89769970a" containerName="init-config-reloader"
Jul 28 10:10:20.171934 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.171803    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc4b0f8e-790d-412e-99fc-bef89769970a" containerName="config-reloader"
Jul 28 10:10:20.171934 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.171808    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc4b0f8e-790d-412e-99fc-bef89769970a" containerName="config-reloader"
Jul 28 10:10:20.171934 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.171815    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc4b0f8e-790d-412e-99fc-bef89769970a" containerName="kube-rbac-proxy-web"
Jul 28 10:10:20.171934 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.171820    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc4b0f8e-790d-412e-99fc-bef89769970a" containerName="kube-rbac-proxy-web"
Jul 28 10:10:20.171934 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.171859    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc4b0f8e-790d-412e-99fc-bef89769970a" containerName="kube-rbac-proxy-web"
Jul 28 10:10:20.171934 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.171866    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4cd4db4-c476-4634-a7ab-ea83fb2ca5ec" containerName="registry"
Jul 28 10:10:20.171934 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.171871    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc4b0f8e-790d-412e-99fc-bef89769970a" containerName="config-reloader"
Jul 28 10:10:20.171934 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.171877    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc4b0f8e-790d-412e-99fc-bef89769970a" containerName="prom-label-proxy"
Jul 28 10:10:20.171934 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.171882    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc4b0f8e-790d-412e-99fc-bef89769970a" containerName="kube-rbac-proxy-metric"
Jul 28 10:10:20.171934 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.171888    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc4b0f8e-790d-412e-99fc-bef89769970a" containerName="alertmanager"
Jul 28 10:10:20.171934 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.171893    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc4b0f8e-790d-412e-99fc-bef89769970a" containerName="kube-rbac-proxy"
Jul 28 10:10:20.175555 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.175539    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:10:20.178429 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.178407    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-pbz2x\""
Jul 28 10:10:20.179885 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.179866    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 28 10:10:20.180120 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.180105    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 28 10:10:20.180699 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.180680    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 28 10:10:20.186565 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.186547    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 28 10:10:20.187639 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.187626    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 28 10:10:20.187839 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.187826    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 28 10:10:20.188210 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.188198    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 28 10:10:20.188806 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.188792    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 28 10:10:20.197723 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.197701    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 28 10:10:20.197902 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.197886    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 10:10:20.263072 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.263032    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/df33571a-1196-4961-8130-351a73ed1a21-tls-assets\") pod \"alertmanager-main-0\" (UID: \"df33571a-1196-4961-8130-351a73ed1a21\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:10:20.263072 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.263084    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/df33571a-1196-4961-8130-351a73ed1a21-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"df33571a-1196-4961-8130-351a73ed1a21\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:10:20.263311 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.263114    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/df33571a-1196-4961-8130-351a73ed1a21-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"df33571a-1196-4961-8130-351a73ed1a21\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:10:20.263311 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.263141    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df33571a-1196-4961-8130-351a73ed1a21-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"df33571a-1196-4961-8130-351a73ed1a21\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:10:20.263311 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.263159    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/df33571a-1196-4961-8130-351a73ed1a21-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"df33571a-1196-4961-8130-351a73ed1a21\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:10:20.263311 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.263236    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/df33571a-1196-4961-8130-351a73ed1a21-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"df33571a-1196-4961-8130-351a73ed1a21\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:10:20.263311 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.263272    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/df33571a-1196-4961-8130-351a73ed1a21-config-volume\") pod \"alertmanager-main-0\" (UID: \"df33571a-1196-4961-8130-351a73ed1a21\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:10:20.263311 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.263293    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/df33571a-1196-4961-8130-351a73ed1a21-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"df33571a-1196-4961-8130-351a73ed1a21\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:10:20.263490 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.263314    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/df33571a-1196-4961-8130-351a73ed1a21-web-config\") pod \"alertmanager-main-0\" (UID: \"df33571a-1196-4961-8130-351a73ed1a21\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:10:20.263490 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.263337    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmj5c\" (UniqueName: \"kubernetes.io/projected/df33571a-1196-4961-8130-351a73ed1a21-kube-api-access-wmj5c\") pod \"alertmanager-main-0\" (UID: \"df33571a-1196-4961-8130-351a73ed1a21\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:10:20.263490 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.263364    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/df33571a-1196-4961-8130-351a73ed1a21-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"df33571a-1196-4961-8130-351a73ed1a21\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:10:20.263490 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.263405    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/df33571a-1196-4961-8130-351a73ed1a21-config-out\") pod \"alertmanager-main-0\" (UID: \"df33571a-1196-4961-8130-351a73ed1a21\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:10:20.263490 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.263431    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/df33571a-1196-4961-8130-351a73ed1a21-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"df33571a-1196-4961-8130-351a73ed1a21\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:10:20.364358 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.364274    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/df33571a-1196-4961-8130-351a73ed1a21-config-out\") pod \"alertmanager-main-0\" (UID: \"df33571a-1196-4961-8130-351a73ed1a21\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:10:20.364358 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.364314    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/df33571a-1196-4961-8130-351a73ed1a21-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"df33571a-1196-4961-8130-351a73ed1a21\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:10:20.364358 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.364335    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/df33571a-1196-4961-8130-351a73ed1a21-tls-assets\") pod \"alertmanager-main-0\" (UID: \"df33571a-1196-4961-8130-351a73ed1a21\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:10:20.364358 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.364356    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/df33571a-1196-4961-8130-351a73ed1a21-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"df33571a-1196-4961-8130-351a73ed1a21\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:10:20.364698 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.364374    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/df33571a-1196-4961-8130-351a73ed1a21-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"df33571a-1196-4961-8130-351a73ed1a21\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:10:20.364698 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.364394    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df33571a-1196-4961-8130-351a73ed1a21-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"df33571a-1196-4961-8130-351a73ed1a21\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:10:20.364698 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.364418    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/df33571a-1196-4961-8130-351a73ed1a21-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"df33571a-1196-4961-8130-351a73ed1a21\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:10:20.364698 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.364576    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/df33571a-1196-4961-8130-351a73ed1a21-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"df33571a-1196-4961-8130-351a73ed1a21\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:10:20.364698 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.364619    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/df33571a-1196-4961-8130-351a73ed1a21-config-volume\") pod \"alertmanager-main-0\" (UID: \"df33571a-1196-4961-8130-351a73ed1a21\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:10:20.364698 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.364642    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/df33571a-1196-4961-8130-351a73ed1a21-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"df33571a-1196-4961-8130-351a73ed1a21\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:10:20.364698 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.364663    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/df33571a-1196-4961-8130-351a73ed1a21-web-config\") pod \"alertmanager-main-0\" (UID: \"df33571a-1196-4961-8130-351a73ed1a21\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:10:20.364698 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.364680    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmj5c\" (UniqueName: \"kubernetes.io/projected/df33571a-1196-4961-8130-351a73ed1a21-kube-api-access-wmj5c\") pod \"alertmanager-main-0\" (UID: \"df33571a-1196-4961-8130-351a73ed1a21\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:10:20.365082 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.364708    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/df33571a-1196-4961-8130-351a73ed1a21-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"df33571a-1196-4961-8130-351a73ed1a21\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:10:20.365082 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.364822    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/df33571a-1196-4961-8130-351a73ed1a21-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"df33571a-1196-4961-8130-351a73ed1a21\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:10:20.365203 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.365110    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/df33571a-1196-4961-8130-351a73ed1a21-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"df33571a-1196-4961-8130-351a73ed1a21\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:10:20.365433 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.365407    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df33571a-1196-4961-8130-351a73ed1a21-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"df33571a-1196-4961-8130-351a73ed1a21\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:10:20.367329 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.367304    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/df33571a-1196-4961-8130-351a73ed1a21-config-out\") pod \"alertmanager-main-0\" (UID: \"df33571a-1196-4961-8130-351a73ed1a21\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:10:20.367436 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.367374    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/df33571a-1196-4961-8130-351a73ed1a21-tls-assets\") pod \"alertmanager-main-0\" (UID: \"df33571a-1196-4961-8130-351a73ed1a21\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:10:20.367635 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.367578    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/df33571a-1196-4961-8130-351a73ed1a21-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"df33571a-1196-4961-8130-351a73ed1a21\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:10:20.367718 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.367644    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/df33571a-1196-4961-8130-351a73ed1a21-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"df33571a-1196-4961-8130-351a73ed1a21\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:10:20.367776 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.367755    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/df33571a-1196-4961-8130-351a73ed1a21-web-config\") pod \"alertmanager-main-0\" (UID: \"df33571a-1196-4961-8130-351a73ed1a21\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:10:20.367942 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.367915    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/df33571a-1196-4961-8130-351a73ed1a21-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"df33571a-1196-4961-8130-351a73ed1a21\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:10:20.368025 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.367945    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/df33571a-1196-4961-8130-351a73ed1a21-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"df33571a-1196-4961-8130-351a73ed1a21\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:10:20.368144 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.368128    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/df33571a-1196-4961-8130-351a73ed1a21-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"df33571a-1196-4961-8130-351a73ed1a21\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:10:20.369249 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.369232    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/df33571a-1196-4961-8130-351a73ed1a21-config-volume\") pod \"alertmanager-main-0\" (UID: \"df33571a-1196-4961-8130-351a73ed1a21\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:10:20.373634 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.373617    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmj5c\" (UniqueName: \"kubernetes.io/projected/df33571a-1196-4961-8130-351a73ed1a21-kube-api-access-wmj5c\") pod \"alertmanager-main-0\" (UID: \"df33571a-1196-4961-8130-351a73ed1a21\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:10:20.484339 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.484301    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:10:20.623110 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:20.623085    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 10:10:20.624603 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:10:20.624578    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf33571a_1196_4961_8130_351a73ed1a21.slice/crio-fdd91adf73e90425299254c460f32c28505b813a9ebaa40ded7012ad00c43327 WatchSource:0}: Error finding container fdd91adf73e90425299254c460f32c28505b813a9ebaa40ded7012ad00c43327: Status 404 returned error can't find the container with id fdd91adf73e90425299254c460f32c28505b813a9ebaa40ded7012ad00c43327
Jul 28 10:10:21.114573 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:21.114538    2583 generic.go:356] "Generic (PLEG): container finished" podID="df33571a-1196-4961-8130-351a73ed1a21" containerID="0d73a8e06f34cd645694eebd2129baaaccbc6196a64f0c6ade3eae45ff20f9d3" exitCode=0
Jul 28 10:10:21.114984 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:21.114626    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"df33571a-1196-4961-8130-351a73ed1a21","Type":"ContainerDied","Data":"0d73a8e06f34cd645694eebd2129baaaccbc6196a64f0c6ade3eae45ff20f9d3"}
Jul 28 10:10:21.114984 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:21.114690    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"df33571a-1196-4961-8130-351a73ed1a21","Type":"ContainerStarted","Data":"fdd91adf73e90425299254c460f32c28505b813a9ebaa40ded7012ad00c43327"}
Jul 28 10:10:21.438750 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:21.438717    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc4b0f8e-790d-412e-99fc-bef89769970a" path="/var/lib/kubelet/pods/bc4b0f8e-790d-412e-99fc-bef89769970a/volumes"
Jul 28 10:10:22.121131 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:22.121089    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"df33571a-1196-4961-8130-351a73ed1a21","Type":"ContainerStarted","Data":"4c0fafb333e8ebc646d873407ae4626ac7007ff1ce964a8b084ef1d93346860d"}
Jul 28 10:10:22.121564 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:22.121210    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"df33571a-1196-4961-8130-351a73ed1a21","Type":"ContainerStarted","Data":"454b7d76e4f974dcdf7a1794c59faf57a1ecd85301b96bc27e97ec6378816ffb"}
Jul 28 10:10:22.121564 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:22.121247    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"df33571a-1196-4961-8130-351a73ed1a21","Type":"ContainerStarted","Data":"6d15508e9d95528992cd21a72ad0af7c5b6bd65c2dc840cdcfc59040e2ab3ff2"}
Jul 28 10:10:22.121564 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:22.121275    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"df33571a-1196-4961-8130-351a73ed1a21","Type":"ContainerStarted","Data":"040f1ae4ee2201e26708c4ff6731020a3222d3ceff15386420aee9a4e4360009"}
Jul 28 10:10:22.121564 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:22.121302    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"df33571a-1196-4961-8130-351a73ed1a21","Type":"ContainerStarted","Data":"3f43d447e577057e94c307653dfeaf10f99e7651c02401172aaf7c84d7221b6e"}
Jul 28 10:10:22.121564 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:22.121344    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"df33571a-1196-4961-8130-351a73ed1a21","Type":"ContainerStarted","Data":"05591d91e6877e287a36f9b0e04124747f7d94058c6f0b5edb9f55d468ae18db"}
Jul 28 10:10:22.148456 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:22.148407    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.132444788 podStartE2EDuration="2.148392362s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.947574ms" isStatefulPod=false podCreationTimestamp="2026-07-28 10:10:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:10:22.146220692 +0000 UTC m=+161.246727246" watchObservedRunningTime="2026-07-28 10:10:22.148392362 +0000 UTC m=+161.248898915"
Jul 28 10:10:22.784168 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:22.784137    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-747b8f97c9-bwl55"]
Jul 28 10:10:22.786648 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:22.786628    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-747b8f97c9-bwl55"
Jul 28 10:10:22.789330 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:22.789309    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 28 10:10:22.789450 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:22.789376    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 28 10:10:22.789835 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:22.789819    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 28 10:10:22.789899 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:22.789848    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 28 10:10:22.789899 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:22.789858    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-phjck\""
Jul 28 10:10:22.790393 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:22.790379    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 28 10:10:22.795591 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:22.795571    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 28 10:10:22.801522 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:22.801501    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-747b8f97c9-bwl55"]
Jul 28 10:10:22.883579 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:22.883556    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/d82bf008-de56-4acb-8005-267b76eea049-telemeter-client-tls\") pod \"telemeter-client-747b8f97c9-bwl55\" (UID: \"d82bf008-de56-4acb-8005-267b76eea049\") " pod="openshift-monitoring/telemeter-client-747b8f97c9-bwl55"
Jul 28 10:10:22.883716 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:22.883584    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/d82bf008-de56-4acb-8005-267b76eea049-federate-client-tls\") pod \"telemeter-client-747b8f97c9-bwl55\" (UID: \"d82bf008-de56-4acb-8005-267b76eea049\") " pod="openshift-monitoring/telemeter-client-747b8f97c9-bwl55"
Jul 28 10:10:22.883716 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:22.883614    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d82bf008-de56-4acb-8005-267b76eea049-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-747b8f97c9-bwl55\" (UID: \"d82bf008-de56-4acb-8005-267b76eea049\") " pod="openshift-monitoring/telemeter-client-747b8f97c9-bwl55"
Jul 28 10:10:22.883716 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:22.883633    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/d82bf008-de56-4acb-8005-267b76eea049-secret-telemeter-client\") pod \"telemeter-client-747b8f97c9-bwl55\" (UID: \"d82bf008-de56-4acb-8005-267b76eea049\") " pod="openshift-monitoring/telemeter-client-747b8f97c9-bwl55"
Jul 28 10:10:22.883716 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:22.883690    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7cqb\" (UniqueName: \"kubernetes.io/projected/d82bf008-de56-4acb-8005-267b76eea049-kube-api-access-d7cqb\") pod \"telemeter-client-747b8f97c9-bwl55\" (UID: \"d82bf008-de56-4acb-8005-267b76eea049\") " pod="openshift-monitoring/telemeter-client-747b8f97c9-bwl55"
Jul 28 10:10:22.883840 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:22.883771    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d82bf008-de56-4acb-8005-267b76eea049-serving-certs-ca-bundle\") pod \"telemeter-client-747b8f97c9-bwl55\" (UID: \"d82bf008-de56-4acb-8005-267b76eea049\") " pod="openshift-monitoring/telemeter-client-747b8f97c9-bwl55"
Jul 28 10:10:22.883840 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:22.883822    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d82bf008-de56-4acb-8005-267b76eea049-telemeter-trusted-ca-bundle\") pod \"telemeter-client-747b8f97c9-bwl55\" (UID: \"d82bf008-de56-4acb-8005-267b76eea049\") " pod="openshift-monitoring/telemeter-client-747b8f97c9-bwl55"
Jul 28 10:10:22.883895 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:22.883844    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d82bf008-de56-4acb-8005-267b76eea049-metrics-client-ca\") pod \"telemeter-client-747b8f97c9-bwl55\" (UID: \"d82bf008-de56-4acb-8005-267b76eea049\") " pod="openshift-monitoring/telemeter-client-747b8f97c9-bwl55"
Jul 28 10:10:22.984955 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:22.984923    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d82bf008-de56-4acb-8005-267b76eea049-telemeter-trusted-ca-bundle\") pod \"telemeter-client-747b8f97c9-bwl55\" (UID: \"d82bf008-de56-4acb-8005-267b76eea049\") " pod="openshift-monitoring/telemeter-client-747b8f97c9-bwl55"
Jul 28 10:10:22.984955 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:22.984958    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d82bf008-de56-4acb-8005-267b76eea049-metrics-client-ca\") pod \"telemeter-client-747b8f97c9-bwl55\" (UID: \"d82bf008-de56-4acb-8005-267b76eea049\") " pod="openshift-monitoring/telemeter-client-747b8f97c9-bwl55"
Jul 28 10:10:22.985159 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:22.984979    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/d82bf008-de56-4acb-8005-267b76eea049-telemeter-client-tls\") pod \"telemeter-client-747b8f97c9-bwl55\" (UID: \"d82bf008-de56-4acb-8005-267b76eea049\") " pod="openshift-monitoring/telemeter-client-747b8f97c9-bwl55"
Jul 28 10:10:22.985159 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:22.984997    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/d82bf008-de56-4acb-8005-267b76eea049-federate-client-tls\") pod \"telemeter-client-747b8f97c9-bwl55\" (UID: \"d82bf008-de56-4acb-8005-267b76eea049\") " pod="openshift-monitoring/telemeter-client-747b8f97c9-bwl55"
Jul 28 10:10:22.985159 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:22.985040    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d82bf008-de56-4acb-8005-267b76eea049-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-747b8f97c9-bwl55\" (UID: \"d82bf008-de56-4acb-8005-267b76eea049\") " pod="openshift-monitoring/telemeter-client-747b8f97c9-bwl55"
Jul 28 10:10:22.985159 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:22.985068    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/d82bf008-de56-4acb-8005-267b76eea049-secret-telemeter-client\") pod \"telemeter-client-747b8f97c9-bwl55\" (UID: \"d82bf008-de56-4acb-8005-267b76eea049\") " pod="openshift-monitoring/telemeter-client-747b8f97c9-bwl55"
Jul 28 10:10:22.985159 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:22.985101    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7cqb\" (UniqueName: \"kubernetes.io/projected/d82bf008-de56-4acb-8005-267b76eea049-kube-api-access-d7cqb\") pod \"telemeter-client-747b8f97c9-bwl55\" (UID: \"d82bf008-de56-4acb-8005-267b76eea049\") " pod="openshift-monitoring/telemeter-client-747b8f97c9-bwl55"
Jul 28 10:10:22.985159 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:22.985151    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d82bf008-de56-4acb-8005-267b76eea049-serving-certs-ca-bundle\") pod \"telemeter-client-747b8f97c9-bwl55\" (UID: \"d82bf008-de56-4acb-8005-267b76eea049\") " pod="openshift-monitoring/telemeter-client-747b8f97c9-bwl55"
Jul 28 10:10:22.985819 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:22.985792    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d82bf008-de56-4acb-8005-267b76eea049-metrics-client-ca\") pod \"telemeter-client-747b8f97c9-bwl55\" (UID: \"d82bf008-de56-4acb-8005-267b76eea049\") " pod="openshift-monitoring/telemeter-client-747b8f97c9-bwl55"
Jul 28 10:10:22.985924 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:22.985857    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d82bf008-de56-4acb-8005-267b76eea049-serving-certs-ca-bundle\") pod \"telemeter-client-747b8f97c9-bwl55\" (UID: \"d82bf008-de56-4acb-8005-267b76eea049\") " pod="openshift-monitoring/telemeter-client-747b8f97c9-bwl55"
Jul 28 10:10:22.985975 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:22.985929    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d82bf008-de56-4acb-8005-267b76eea049-telemeter-trusted-ca-bundle\") pod \"telemeter-client-747b8f97c9-bwl55\" (UID: \"d82bf008-de56-4acb-8005-267b76eea049\") " pod="openshift-monitoring/telemeter-client-747b8f97c9-bwl55"
Jul 28 10:10:22.987685 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:22.987663    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/d82bf008-de56-4acb-8005-267b76eea049-secret-telemeter-client\") pod \"telemeter-client-747b8f97c9-bwl55\" (UID: \"d82bf008-de56-4acb-8005-267b76eea049\") " pod="openshift-monitoring/telemeter-client-747b8f97c9-bwl55"
Jul 28 10:10:22.987882 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:22.987864    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/d82bf008-de56-4acb-8005-267b76eea049-federate-client-tls\") pod \"telemeter-client-747b8f97c9-bwl55\" (UID: \"d82bf008-de56-4acb-8005-267b76eea049\") " pod="openshift-monitoring/telemeter-client-747b8f97c9-bwl55"
Jul 28 10:10:22.988013 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:22.987991    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d82bf008-de56-4acb-8005-267b76eea049-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-747b8f97c9-bwl55\" (UID: \"d82bf008-de56-4acb-8005-267b76eea049\") " pod="openshift-monitoring/telemeter-client-747b8f97c9-bwl55"
Jul 28 10:10:22.988057 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:22.988041    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/d82bf008-de56-4acb-8005-267b76eea049-telemeter-client-tls\") pod \"telemeter-client-747b8f97c9-bwl55\" (UID: \"d82bf008-de56-4acb-8005-267b76eea049\") " pod="openshift-monitoring/telemeter-client-747b8f97c9-bwl55"
Jul 28 10:10:22.992865 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:22.992847    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7cqb\" (UniqueName: \"kubernetes.io/projected/d82bf008-de56-4acb-8005-267b76eea049-kube-api-access-d7cqb\") pod \"telemeter-client-747b8f97c9-bwl55\" (UID: \"d82bf008-de56-4acb-8005-267b76eea049\") " pod="openshift-monitoring/telemeter-client-747b8f97c9-bwl55"
Jul 28 10:10:23.096361 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:23.096289    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-747b8f97c9-bwl55"
Jul 28 10:10:23.225985 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:23.225960    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-747b8f97c9-bwl55"]
Jul 28 10:10:23.227674 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:10:23.227645    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd82bf008_de56_4acb_8005_267b76eea049.slice/crio-62071506be5cd42105df1de45701201d47819ce646a4ebe03b6b4dc892ecac26 WatchSource:0}: Error finding container 62071506be5cd42105df1de45701201d47819ce646a4ebe03b6b4dc892ecac26: Status 404 returned error can't find the container with id 62071506be5cd42105df1de45701201d47819ce646a4ebe03b6b4dc892ecac26
Jul 28 10:10:24.130325 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:24.130285    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-747b8f97c9-bwl55" event={"ID":"d82bf008-de56-4acb-8005-267b76eea049","Type":"ContainerStarted","Data":"62071506be5cd42105df1de45701201d47819ce646a4ebe03b6b4dc892ecac26"}
Jul 28 10:10:26.141113 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:26.141028    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-747b8f97c9-bwl55" event={"ID":"d82bf008-de56-4acb-8005-267b76eea049","Type":"ContainerStarted","Data":"f8e919e164a3644113fce7694fc8204dfff598a5254f7b9e067a65c69e4298e5"}
Jul 28 10:10:26.141113 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:26.141099    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-747b8f97c9-bwl55" event={"ID":"d82bf008-de56-4acb-8005-267b76eea049","Type":"ContainerStarted","Data":"d38241457c0d85b68a1d5585477e85149d48f545dad7e09e6a65c20bc816fb22"}
Jul 28 10:10:26.141522 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:26.141128    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-747b8f97c9-bwl55" event={"ID":"d82bf008-de56-4acb-8005-267b76eea049","Type":"ContainerStarted","Data":"421e229997fdf355de573cd4d10450467cb1f25fdd12e464f823bc969c637a92"}
Jul 28 10:10:26.163537 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:26.163489    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-747b8f97c9-bwl55" podStartSLOduration=1.6774189480000001 podStartE2EDuration="4.16347536s" totalImagesPullingTime="2.486056412s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:10:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:10:26.163003886 +0000 UTC m=+165.263510453" watchObservedRunningTime="2026-07-28 10:10:26.16347536 +0000 UTC m=+165.263981913"
Jul 28 10:10:27.077467 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:27.077428    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:10:27.177821 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:27.177787    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:10:27.278647 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:27.278620    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:10:27.806582 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:27.806550    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:10:27.907126 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:27.907096    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:10:38.201985 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:38.201901    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:10:38.301318 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:38.301288    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:10:38.402103 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:38.402073    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:10:38.466819 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:10:38.466742    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:11:32.932360 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:11:32.932325    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:11:32.994516 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:11:32.994483    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:11:35.325398 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:11:35.325367    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:11:44.915667 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:11:44.915633    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:11:44.982637 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:11:44.982610    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:12:00.374744 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:12:00.374711    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:12:02.252498 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:12:02.252456    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:12:02.350526 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:12:02.350501    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:12:06.804030 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:12:06.803959    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:12:06.877986 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:12:06.877962    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:12:07.125845 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:12:07.125779    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:12:07.903775 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:12:07.903747    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:12:14.307751 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:12:14.307707    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:12:17.870245 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:12:17.870210    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:12:17.939346 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:12:17.939317    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:12:41.261025 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:12:41.260984    2583 kubelet.go:1648] "Image garbage collection succeeded"
Jul 28 10:13:38.219952 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:13:38.219875    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:13:39.381300 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:13:39.381259    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:13:39.524094 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:13:39.524064    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5k9k6d"]
Jul 28 10:13:39.527297 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:13:39.527276    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5k9k6d"
Jul 28 10:13:39.529676 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:13:39.529656    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 28 10:13:39.529676 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:13:39.529662    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 28 10:13:39.530170 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:13:39.530154    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-fqqtw\""
Jul 28 10:13:39.537531 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:13:39.537510    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5k9k6d"]
Jul 28 10:13:39.588152 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:13:39.588131    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9ac4fa68-b830-49d1-ba1c-b501d0343403-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5k9k6d\" (UID: \"9ac4fa68-b830-49d1-ba1c-b501d0343403\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5k9k6d"
Jul 28 10:13:39.588278 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:13:39.588172    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9ac4fa68-b830-49d1-ba1c-b501d0343403-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5k9k6d\" (UID: \"9ac4fa68-b830-49d1-ba1c-b501d0343403\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5k9k6d"
Jul 28 10:13:39.588345 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:13:39.588274    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8bnq\" (UniqueName: \"kubernetes.io/projected/9ac4fa68-b830-49d1-ba1c-b501d0343403-kube-api-access-q8bnq\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5k9k6d\" (UID: \"9ac4fa68-b830-49d1-ba1c-b501d0343403\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5k9k6d"
Jul 28 10:13:39.688590 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:13:39.688567    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9ac4fa68-b830-49d1-ba1c-b501d0343403-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5k9k6d\" (UID: \"9ac4fa68-b830-49d1-ba1c-b501d0343403\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5k9k6d"
Jul 28 10:13:39.688692 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:13:39.688607    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8bnq\" (UniqueName: \"kubernetes.io/projected/9ac4fa68-b830-49d1-ba1c-b501d0343403-kube-api-access-q8bnq\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5k9k6d\" (UID: \"9ac4fa68-b830-49d1-ba1c-b501d0343403\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5k9k6d"
Jul 28 10:13:39.688692 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:13:39.688639    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9ac4fa68-b830-49d1-ba1c-b501d0343403-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5k9k6d\" (UID: \"9ac4fa68-b830-49d1-ba1c-b501d0343403\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5k9k6d"
Jul 28 10:13:39.688984 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:13:39.688960    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9ac4fa68-b830-49d1-ba1c-b501d0343403-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5k9k6d\" (UID: \"9ac4fa68-b830-49d1-ba1c-b501d0343403\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5k9k6d"
Jul 28 10:13:39.689040 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:13:39.688971    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9ac4fa68-b830-49d1-ba1c-b501d0343403-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5k9k6d\" (UID: \"9ac4fa68-b830-49d1-ba1c-b501d0343403\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5k9k6d"
Jul 28 10:13:39.697588 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:13:39.697561    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8bnq\" (UniqueName: \"kubernetes.io/projected/9ac4fa68-b830-49d1-ba1c-b501d0343403-kube-api-access-q8bnq\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5k9k6d\" (UID: \"9ac4fa68-b830-49d1-ba1c-b501d0343403\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5k9k6d"
Jul 28 10:13:39.836206 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:13:39.836168    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5k9k6d"
Jul 28 10:13:39.963850 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:13:39.963827    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5k9k6d"]
Jul 28 10:13:39.966317 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:13:39.966290    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ac4fa68_b830_49d1_ba1c_b501d0343403.slice/crio-91714761b88d37a76a87e07251859d0bf7a1eb5f286f6ef4fea8396a7616100e WatchSource:0}: Error finding container 91714761b88d37a76a87e07251859d0bf7a1eb5f286f6ef4fea8396a7616100e: Status 404 returned error can't find the container with id 91714761b88d37a76a87e07251859d0bf7a1eb5f286f6ef4fea8396a7616100e
Jul 28 10:13:39.968296 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:13:39.968277    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 10:13:40.846250 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:13:40.846213    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5k9k6d" event={"ID":"9ac4fa68-b830-49d1-ba1c-b501d0343403","Type":"ContainerStarted","Data":"91714761b88d37a76a87e07251859d0bf7a1eb5f286f6ef4fea8396a7616100e"}
Jul 28 10:13:46.871742 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:13:46.871709    2583 generic.go:356] "Generic (PLEG): container finished" podID="9ac4fa68-b830-49d1-ba1c-b501d0343403" containerID="0a1d65de0e0f215bfd73fe12131019236657b7c4ad114118a71bf092ee40a8bb" exitCode=0
Jul 28 10:13:46.872167 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:13:46.871792    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5k9k6d" event={"ID":"9ac4fa68-b830-49d1-ba1c-b501d0343403","Type":"ContainerDied","Data":"0a1d65de0e0f215bfd73fe12131019236657b7c4ad114118a71bf092ee40a8bb"}
Jul 28 10:13:48.881716 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:13:48.881687    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5k9k6d" event={"ID":"9ac4fa68-b830-49d1-ba1c-b501d0343403","Type":"ContainerStarted","Data":"3497d7d9968dea4366f31d8ba118fe36c1aca18e4da25faa33ffa276bf3a1d16"}
Jul 28 10:13:49.887266 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:13:49.887230    2583 generic.go:356] "Generic (PLEG): container finished" podID="9ac4fa68-b830-49d1-ba1c-b501d0343403" containerID="3497d7d9968dea4366f31d8ba118fe36c1aca18e4da25faa33ffa276bf3a1d16" exitCode=0
Jul 28 10:13:49.887727 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:13:49.887319    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5k9k6d" event={"ID":"9ac4fa68-b830-49d1-ba1c-b501d0343403","Type":"ContainerDied","Data":"3497d7d9968dea4366f31d8ba118fe36c1aca18e4da25faa33ffa276bf3a1d16"}
Jul 28 10:13:56.916581 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:13:56.916548    2583 generic.go:356] "Generic (PLEG): container finished" podID="9ac4fa68-b830-49d1-ba1c-b501d0343403" containerID="236e3d666e31de96e0d17c7a7b99486751a561bde8012a32bcffb2233fda6bbb" exitCode=0
Jul 28 10:13:56.916988 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:13:56.916616    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5k9k6d" event={"ID":"9ac4fa68-b830-49d1-ba1c-b501d0343403","Type":"ContainerDied","Data":"236e3d666e31de96e0d17c7a7b99486751a561bde8012a32bcffb2233fda6bbb"}
Jul 28 10:13:58.046252 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:13:58.046229    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5k9k6d"
Jul 28 10:13:58.150614 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:13:58.150580    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q8bnq\" (UniqueName: \"kubernetes.io/projected/9ac4fa68-b830-49d1-ba1c-b501d0343403-kube-api-access-q8bnq\") pod \"9ac4fa68-b830-49d1-ba1c-b501d0343403\" (UID: \"9ac4fa68-b830-49d1-ba1c-b501d0343403\") "
Jul 28 10:13:58.150775 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:13:58.150630    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9ac4fa68-b830-49d1-ba1c-b501d0343403-util\") pod \"9ac4fa68-b830-49d1-ba1c-b501d0343403\" (UID: \"9ac4fa68-b830-49d1-ba1c-b501d0343403\") "
Jul 28 10:13:58.150775 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:13:58.150712    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9ac4fa68-b830-49d1-ba1c-b501d0343403-bundle\") pod \"9ac4fa68-b830-49d1-ba1c-b501d0343403\" (UID: \"9ac4fa68-b830-49d1-ba1c-b501d0343403\") "
Jul 28 10:13:58.151384 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:13:58.151356    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ac4fa68-b830-49d1-ba1c-b501d0343403-bundle" (OuterVolumeSpecName: "bundle") pod "9ac4fa68-b830-49d1-ba1c-b501d0343403" (UID: "9ac4fa68-b830-49d1-ba1c-b501d0343403"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:13:58.152812 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:13:58.152787    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ac4fa68-b830-49d1-ba1c-b501d0343403-kube-api-access-q8bnq" (OuterVolumeSpecName: "kube-api-access-q8bnq") pod "9ac4fa68-b830-49d1-ba1c-b501d0343403" (UID: "9ac4fa68-b830-49d1-ba1c-b501d0343403"). InnerVolumeSpecName "kube-api-access-q8bnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:13:58.154802 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:13:58.154778    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ac4fa68-b830-49d1-ba1c-b501d0343403-util" (OuterVolumeSpecName: "util") pod "9ac4fa68-b830-49d1-ba1c-b501d0343403" (UID: "9ac4fa68-b830-49d1-ba1c-b501d0343403"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:13:58.252151 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:13:58.252073    2583 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9ac4fa68-b830-49d1-ba1c-b501d0343403-bundle\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:13:58.252151 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:13:58.252100    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8bnq\" (UniqueName: \"kubernetes.io/projected/9ac4fa68-b830-49d1-ba1c-b501d0343403-kube-api-access-q8bnq\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:13:58.252151 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:13:58.252111    2583 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9ac4fa68-b830-49d1-ba1c-b501d0343403-util\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:13:58.925620 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:13:58.925587    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5k9k6d" event={"ID":"9ac4fa68-b830-49d1-ba1c-b501d0343403","Type":"ContainerDied","Data":"91714761b88d37a76a87e07251859d0bf7a1eb5f286f6ef4fea8396a7616100e"}
Jul 28 10:13:58.925620 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:13:58.925627    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5k9k6d"
Jul 28 10:13:58.925833 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:13:58.925634    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91714761b88d37a76a87e07251859d0bf7a1eb5f286f6ef4fea8396a7616100e"
Jul 28 10:14:00.694727 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:14:00.694693    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:14:00.804589 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:14:00.804557    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:14:02.103322 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:14:02.103294    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-2hq8q"]
Jul 28 10:14:02.103683 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:14:02.103583    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ac4fa68-b830-49d1-ba1c-b501d0343403" containerName="pull"
Jul 28 10:14:02.103683 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:14:02.103594    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ac4fa68-b830-49d1-ba1c-b501d0343403" containerName="pull"
Jul 28 10:14:02.103683 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:14:02.103603    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ac4fa68-b830-49d1-ba1c-b501d0343403" containerName="util"
Jul 28 10:14:02.103683 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:14:02.103608    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ac4fa68-b830-49d1-ba1c-b501d0343403" containerName="util"
Jul 28 10:14:02.103683 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:14:02.103613    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ac4fa68-b830-49d1-ba1c-b501d0343403" containerName="extract"
Jul 28 10:14:02.103683 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:14:02.103618    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ac4fa68-b830-49d1-ba1c-b501d0343403" containerName="extract"
Jul 28 10:14:02.103683 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:14:02.103671    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ac4fa68-b830-49d1-ba1c-b501d0343403" containerName="extract"
Jul 28 10:14:02.139034 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:14:02.139004    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-2hq8q"]
Jul 28 10:14:02.139201 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:14:02.139115    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-2hq8q"
Jul 28 10:14:02.141489 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:14:02.141472    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 28 10:14:02.141604 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:14:02.141531    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 28 10:14:02.141653 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:14:02.141611    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 28 10:14:02.141653 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:14:02.141634    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-tdfq4\""
Jul 28 10:14:02.179555 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:14:02.179521    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9kxp\" (UniqueName: \"kubernetes.io/projected/1197aecd-0c94-4f7a-b632-e42b4a6f2941-kube-api-access-c9kxp\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-2hq8q\" (UID: \"1197aecd-0c94-4f7a-b632-e42b4a6f2941\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-2hq8q"
Jul 28 10:14:02.179694 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:14:02.179564    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/1197aecd-0c94-4f7a-b632-e42b4a6f2941-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-2hq8q\" (UID: \"1197aecd-0c94-4f7a-b632-e42b4a6f2941\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-2hq8q"
Jul 28 10:14:02.280286 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:14:02.280261    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9kxp\" (UniqueName: \"kubernetes.io/projected/1197aecd-0c94-4f7a-b632-e42b4a6f2941-kube-api-access-c9kxp\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-2hq8q\" (UID: \"1197aecd-0c94-4f7a-b632-e42b4a6f2941\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-2hq8q"
Jul 28 10:14:02.280427 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:14:02.280292    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/1197aecd-0c94-4f7a-b632-e42b4a6f2941-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-2hq8q\" (UID: \"1197aecd-0c94-4f7a-b632-e42b4a6f2941\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-2hq8q"
Jul 28 10:14:02.282475 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:14:02.282451    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/1197aecd-0c94-4f7a-b632-e42b4a6f2941-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-2hq8q\" (UID: \"1197aecd-0c94-4f7a-b632-e42b4a6f2941\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-2hq8q"
Jul 28 10:14:02.288607 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:14:02.288586    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9kxp\" (UniqueName: \"kubernetes.io/projected/1197aecd-0c94-4f7a-b632-e42b4a6f2941-kube-api-access-c9kxp\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-2hq8q\" (UID: \"1197aecd-0c94-4f7a-b632-e42b4a6f2941\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-2hq8q"
Jul 28 10:14:02.448929 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:14:02.448902    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-2hq8q"
Jul 28 10:14:02.597746 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:14:02.597715    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-2hq8q"]
Jul 28 10:14:02.600112 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:14:02.600086    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1197aecd_0c94_4f7a_b632_e42b4a6f2941.slice/crio-68f5a51c0a53c59db613803f8bd95454649e2a2673e955ee4e91acb62f792c7b WatchSource:0}: Error finding container 68f5a51c0a53c59db613803f8bd95454649e2a2673e955ee4e91acb62f792c7b: Status 404 returned error can't find the container with id 68f5a51c0a53c59db613803f8bd95454649e2a2673e955ee4e91acb62f792c7b
Jul 28 10:14:02.948143 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:14:02.948109    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-2hq8q" event={"ID":"1197aecd-0c94-4f7a-b632-e42b4a6f2941","Type":"ContainerStarted","Data":"68f5a51c0a53c59db613803f8bd95454649e2a2673e955ee4e91acb62f792c7b"}
Jul 28 10:14:05.964295 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:14:05.964268    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-2hq8q" event={"ID":"1197aecd-0c94-4f7a-b632-e42b4a6f2941","Type":"ContainerStarted","Data":"25ebb7f2d51fc480fb2a61cf4c34392f3d97a5b07fea49cfddea5908d763d957"}
Jul 28 10:14:05.964634 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:14:05.964404    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-2hq8q"
Jul 28 10:14:05.984600 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:14:05.984554    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-2hq8q" podStartSLOduration=0.743933173 podStartE2EDuration="3.984539321s" totalImagesPullingTime="3.240606148s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:14:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:14:05.982961681 +0000 UTC m=+385.083468257" watchObservedRunningTime="2026-07-28 10:14:05.984539321 +0000 UTC m=+385.085045874"
Jul 28 10:14:06.311388 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:14:06.311345    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:14:06.589024 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:14:06.588952    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:14:06.688267 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:14:06.688233    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:14:06.796285 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:14:06.796251    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:14:07.005089 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:14:07.005060    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:14:07.104502 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:14:07.104475    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:14:07.204356 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:14:07.204313    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:14:07.506208 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:14:07.506165    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:14:07.763085 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:14:07.763014    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:14:07.944650 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:14:07.944613    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:14:08.055820 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:14:08.055752    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:14:26.971168 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:14:26.971135    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-2hq8q"
Jul 28 10:15:26.224600 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:15:26.224569    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:15:26.297864 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:15:26.297827    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:15:26.307018 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:15:26.306989    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-654c557856-tpx8s"]
Jul 28 10:15:26.309763 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:15:26.309742    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-654c557856-tpx8s"
Jul 28 10:15:26.311976 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:15:26.311954    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 28 10:15:26.312096 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:15:26.311954    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-lddbn\""
Jul 28 10:15:26.312511 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:15:26.312487    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 28 10:15:26.312634 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:15:26.312516    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Jul 28 10:15:26.321282 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:15:26.321261    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-654c557856-tpx8s"]
Jul 28 10:15:26.328950 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:15:26.328923    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj9wg\" (UniqueName: \"kubernetes.io/projected/772e05b4-f441-49be-9733-dd14f213ddf6-kube-api-access-vj9wg\") pod \"llmisvc-controller-manager-654c557856-tpx8s\" (UID: \"772e05b4-f441-49be-9733-dd14f213ddf6\") " pod="kserve/llmisvc-controller-manager-654c557856-tpx8s"
Jul 28 10:15:26.329046 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:15:26.328974    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/772e05b4-f441-49be-9733-dd14f213ddf6-cert\") pod \"llmisvc-controller-manager-654c557856-tpx8s\" (UID: \"772e05b4-f441-49be-9733-dd14f213ddf6\") " pod="kserve/llmisvc-controller-manager-654c557856-tpx8s"
Jul 28 10:15:26.358049 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:15:26.358025    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:15:26.430408 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:15:26.430382    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj9wg\" (UniqueName: \"kubernetes.io/projected/772e05b4-f441-49be-9733-dd14f213ddf6-kube-api-access-vj9wg\") pod \"llmisvc-controller-manager-654c557856-tpx8s\" (UID: \"772e05b4-f441-49be-9733-dd14f213ddf6\") " pod="kserve/llmisvc-controller-manager-654c557856-tpx8s"
Jul 28 10:15:26.430494 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:15:26.430431    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/772e05b4-f441-49be-9733-dd14f213ddf6-cert\") pod \"llmisvc-controller-manager-654c557856-tpx8s\" (UID: \"772e05b4-f441-49be-9733-dd14f213ddf6\") " pod="kserve/llmisvc-controller-manager-654c557856-tpx8s"
Jul 28 10:15:26.430596 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:15:26.430576    2583 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Jul 28 10:15:26.430667 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:15:26.430655    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/772e05b4-f441-49be-9733-dd14f213ddf6-cert podName:772e05b4-f441-49be-9733-dd14f213ddf6 nodeName:}" failed. No retries permitted until 2026-07-28 10:15:26.930635819 +0000 UTC m=+466.031142364 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/772e05b4-f441-49be-9733-dd14f213ddf6-cert") pod "llmisvc-controller-manager-654c557856-tpx8s" (UID: "772e05b4-f441-49be-9733-dd14f213ddf6") : secret "llmisvc-webhook-server-cert" not found
Jul 28 10:15:26.441855 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:15:26.441832    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj9wg\" (UniqueName: \"kubernetes.io/projected/772e05b4-f441-49be-9733-dd14f213ddf6-kube-api-access-vj9wg\") pod \"llmisvc-controller-manager-654c557856-tpx8s\" (UID: \"772e05b4-f441-49be-9733-dd14f213ddf6\") " pod="kserve/llmisvc-controller-manager-654c557856-tpx8s"
Jul 28 10:15:26.459300 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:15:26.459271    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:15:26.558492 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:15:26.558439    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:15:26.658988 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:15:26.658962    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:15:26.812560 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:15:26.812499    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:15:26.914429 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:15:26.914395    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:15:26.934702 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:15:26.934682    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/772e05b4-f441-49be-9733-dd14f213ddf6-cert\") pod \"llmisvc-controller-manager-654c557856-tpx8s\" (UID: \"772e05b4-f441-49be-9733-dd14f213ddf6\") " pod="kserve/llmisvc-controller-manager-654c557856-tpx8s"
Jul 28 10:15:26.936939 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:15:26.936923    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/772e05b4-f441-49be-9733-dd14f213ddf6-cert\") pod \"llmisvc-controller-manager-654c557856-tpx8s\" (UID: \"772e05b4-f441-49be-9733-dd14f213ddf6\") " pod="kserve/llmisvc-controller-manager-654c557856-tpx8s"
Jul 28 10:15:26.975149 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:15:26.975133    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:15:27.221700 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:15:27.221677    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-654c557856-tpx8s"
Jul 28 10:15:27.348436 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:15:27.348407    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-654c557856-tpx8s"]
Jul 28 10:15:27.351633 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:15:27.351606    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod772e05b4_f441_49be_9733_dd14f213ddf6.slice/crio-0705dfde76cc2e0816746f437cddb5d7ae184e1f21e79c25cd1c280122ea9d1a WatchSource:0}: Error finding container 0705dfde76cc2e0816746f437cddb5d7ae184e1f21e79c25cd1c280122ea9d1a: Status 404 returned error can't find the container with id 0705dfde76cc2e0816746f437cddb5d7ae184e1f21e79c25cd1c280122ea9d1a
Jul 28 10:15:27.454920 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:15:27.454897    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:15:27.588540 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:15:27.588485    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:15:27.639859 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:15:27.639838    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:15:27.986900 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:15:27.986869    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:15:28.055800 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:15:28.055772    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:15:28.318017 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:15:28.317945    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-654c557856-tpx8s" event={"ID":"772e05b4-f441-49be-9733-dd14f213ddf6","Type":"ContainerStarted","Data":"0705dfde76cc2e0816746f437cddb5d7ae184e1f21e79c25cd1c280122ea9d1a"}
Jul 28 10:15:30.329208 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:15:30.329155    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-654c557856-tpx8s" event={"ID":"772e05b4-f441-49be-9733-dd14f213ddf6","Type":"ContainerStarted","Data":"0a42ceed822e3891e13ee465e38c367c5fafdf4ca8efeaac5a064f606b193a66"}
Jul 28 10:15:30.329624 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:15:30.329260    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-654c557856-tpx8s"
Jul 28 10:15:30.345509 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:15:30.345462    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-654c557856-tpx8s" podStartSLOduration=1.926004598 podStartE2EDuration="4.345446817s" totalImagesPullingTime="2.419442219s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:15:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:15:30.344038893 +0000 UTC m=+469.444545447" watchObservedRunningTime="2026-07-28 10:15:30.345446817 +0000 UTC m=+469.445953360"
Jul 28 10:15:30.479520 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:15:30.479494    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:15:30.579677 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:15:30.579617    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:15:38.383062 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:15:38.383028    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:15:38.439342 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:15:38.439308    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:15:38.497628 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:15:38.497601    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:16:01.335350 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:16:01.335316    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-654c557856-tpx8s"
Jul 28 10:16:11.249106 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:16:11.249073    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:16:11.303103 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:16:11.303078    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:16:43.339469 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:16:43.339368    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:16:43.378176 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:16:43.378140    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-jsdb5"]
Jul 28 10:16:43.381455 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:16:43.381429    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-jsdb5"
Jul 28 10:16:43.384038 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:16:43.384013    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 28 10:16:43.384038 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:16:43.384033    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-67klq\""
Jul 28 10:16:43.402167 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:16:43.402136    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-jsdb5"]
Jul 28 10:16:43.416403 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:16:43.416377    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98p2d\" (UniqueName: \"kubernetes.io/projected/c3a1f9c4-3ef5-4c8e-88db-526ed435b7cf-kube-api-access-98p2d\") pod \"model-serving-api-7b954ffc95-jsdb5\" (UID: \"c3a1f9c4-3ef5-4c8e-88db-526ed435b7cf\") " pod="kserve/model-serving-api-7b954ffc95-jsdb5"
Jul 28 10:16:43.416493 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:16:43.416431    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3a1f9c4-3ef5-4c8e-88db-526ed435b7cf-tls-certs\") pod \"model-serving-api-7b954ffc95-jsdb5\" (UID: \"c3a1f9c4-3ef5-4c8e-88db-526ed435b7cf\") " pod="kserve/model-serving-api-7b954ffc95-jsdb5"
Jul 28 10:16:43.430145 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:16:43.430121    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:16:43.517534 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:16:43.517506    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98p2d\" (UniqueName: \"kubernetes.io/projected/c3a1f9c4-3ef5-4c8e-88db-526ed435b7cf-kube-api-access-98p2d\") pod \"model-serving-api-7b954ffc95-jsdb5\" (UID: \"c3a1f9c4-3ef5-4c8e-88db-526ed435b7cf\") " pod="kserve/model-serving-api-7b954ffc95-jsdb5"
Jul 28 10:16:43.517626 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:16:43.517558    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3a1f9c4-3ef5-4c8e-88db-526ed435b7cf-tls-certs\") pod \"model-serving-api-7b954ffc95-jsdb5\" (UID: \"c3a1f9c4-3ef5-4c8e-88db-526ed435b7cf\") " pod="kserve/model-serving-api-7b954ffc95-jsdb5"
Jul 28 10:16:43.517683 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:16:43.517670    2583 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
Jul 28 10:16:43.517740 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:16:43.517730    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3a1f9c4-3ef5-4c8e-88db-526ed435b7cf-tls-certs podName:c3a1f9c4-3ef5-4c8e-88db-526ed435b7cf nodeName:}" failed. No retries permitted until 2026-07-28 10:16:44.017712035 +0000 UTC m=+543.118218574 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/c3a1f9c4-3ef5-4c8e-88db-526ed435b7cf-tls-certs") pod "model-serving-api-7b954ffc95-jsdb5" (UID: "c3a1f9c4-3ef5-4c8e-88db-526ed435b7cf") : secret "model-serving-api-tls" not found
Jul 28 10:16:43.526324 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:16:43.526300    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98p2d\" (UniqueName: \"kubernetes.io/projected/c3a1f9c4-3ef5-4c8e-88db-526ed435b7cf-kube-api-access-98p2d\") pod \"model-serving-api-7b954ffc95-jsdb5\" (UID: \"c3a1f9c4-3ef5-4c8e-88db-526ed435b7cf\") " pod="kserve/model-serving-api-7b954ffc95-jsdb5"
Jul 28 10:16:43.812446 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:16:43.812410    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:16:44.022377 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:16:44.022346    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3a1f9c4-3ef5-4c8e-88db-526ed435b7cf-tls-certs\") pod \"model-serving-api-7b954ffc95-jsdb5\" (UID: \"c3a1f9c4-3ef5-4c8e-88db-526ed435b7cf\") " pod="kserve/model-serving-api-7b954ffc95-jsdb5"
Jul 28 10:16:44.022527 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:16:44.022487    2583 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
Jul 28 10:16:44.022575 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:16:44.022547    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3a1f9c4-3ef5-4c8e-88db-526ed435b7cf-tls-certs podName:c3a1f9c4-3ef5-4c8e-88db-526ed435b7cf nodeName:}" failed. No retries permitted until 2026-07-28 10:16:45.022532324 +0000 UTC m=+544.123038857 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/c3a1f9c4-3ef5-4c8e-88db-526ed435b7cf-tls-certs") pod "model-serving-api-7b954ffc95-jsdb5" (UID: "c3a1f9c4-3ef5-4c8e-88db-526ed435b7cf") : secret "model-serving-api-tls" not found
Jul 28 10:16:44.060111 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:16:44.060081    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:16:44.115106 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:16:44.115044    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:16:44.539096 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:16:44.539070    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:16:45.029952 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:16:45.029923    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3a1f9c4-3ef5-4c8e-88db-526ed435b7cf-tls-certs\") pod \"model-serving-api-7b954ffc95-jsdb5\" (UID: \"c3a1f9c4-3ef5-4c8e-88db-526ed435b7cf\") " pod="kserve/model-serving-api-7b954ffc95-jsdb5"
Jul 28 10:16:45.032276 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:16:45.032253    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3a1f9c4-3ef5-4c8e-88db-526ed435b7cf-tls-certs\") pod \"model-serving-api-7b954ffc95-jsdb5\" (UID: \"c3a1f9c4-3ef5-4c8e-88db-526ed435b7cf\") " pod="kserve/model-serving-api-7b954ffc95-jsdb5"
Jul 28 10:16:45.192496 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:16:45.192470    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-jsdb5"
Jul 28 10:16:45.319637 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:16:45.319521    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-jsdb5"]
Jul 28 10:16:45.322268 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:16:45.322237    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3a1f9c4_3ef5_4c8e_88db_526ed435b7cf.slice/crio-37762e5d76136c6eac00b504559a6c8ff25f6bc7088567d92420fadc5465f30c WatchSource:0}: Error finding container 37762e5d76136c6eac00b504559a6c8ff25f6bc7088567d92420fadc5465f30c: Status 404 returned error can't find the container with id 37762e5d76136c6eac00b504559a6c8ff25f6bc7088567d92420fadc5465f30c
Jul 28 10:16:45.636029 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:16:45.635957    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-jsdb5" event={"ID":"c3a1f9c4-3ef5-4c8e-88db-526ed435b7cf","Type":"ContainerStarted","Data":"37762e5d76136c6eac00b504559a6c8ff25f6bc7088567d92420fadc5465f30c"}
Jul 28 10:16:47.648085 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:16:47.648052    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-jsdb5" event={"ID":"c3a1f9c4-3ef5-4c8e-88db-526ed435b7cf","Type":"ContainerStarted","Data":"1da6d92f55efb6b0f3349a71a5028ec8d90a09c12e0e5ece8d577000281438e4"}
Jul 28 10:16:47.648438 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:16:47.648132    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-jsdb5"
Jul 28 10:16:47.668410 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:16:47.668355    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-jsdb5" podStartSLOduration=2.431398814 podStartE2EDuration="4.668340125s" totalImagesPullingTime="2.236941311s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:16:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:16:47.666686722 +0000 UTC m=+546.767193277" watchObservedRunningTime="2026-07-28 10:16:47.668340125 +0000 UTC m=+546.768846679"
Jul 28 10:16:58.660674 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:16:58.660644    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-jsdb5"
Jul 28 10:16:59.368986 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:16:59.368947    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-lq5rg"]
Jul 28 10:16:59.372470 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:16:59.372454    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-lq5rg"
Jul 28 10:16:59.374896 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:16:59.374874    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 28 10:16:59.374896 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:16:59.374893    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-2wxxj\""
Jul 28 10:16:59.381076 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:16:59.381051    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-lq5rg"]
Jul 28 10:16:59.452933 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:16:59.452902    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b46c2c15-92be-452b-9fd7-0ce038fbc9d9-model-cache\") pod \"s3-init-lq5rg\" (UID: \"b46c2c15-92be-452b-9fd7-0ce038fbc9d9\") " pod="kserve/s3-init-lq5rg"
Jul 28 10:16:59.452933 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:16:59.452935    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftgsz\" (UniqueName: \"kubernetes.io/projected/b46c2c15-92be-452b-9fd7-0ce038fbc9d9-kube-api-access-ftgsz\") pod \"s3-init-lq5rg\" (UID: \"b46c2c15-92be-452b-9fd7-0ce038fbc9d9\") " pod="kserve/s3-init-lq5rg"
Jul 28 10:16:59.554221 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:16:59.554166    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b46c2c15-92be-452b-9fd7-0ce038fbc9d9-model-cache\") pod \"s3-init-lq5rg\" (UID: \"b46c2c15-92be-452b-9fd7-0ce038fbc9d9\") " pod="kserve/s3-init-lq5rg"
Jul 28 10:16:59.554372 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:16:59.554226    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftgsz\" (UniqueName: \"kubernetes.io/projected/b46c2c15-92be-452b-9fd7-0ce038fbc9d9-kube-api-access-ftgsz\") pod \"s3-init-lq5rg\" (UID: \"b46c2c15-92be-452b-9fd7-0ce038fbc9d9\") " pod="kserve/s3-init-lq5rg"
Jul 28 10:16:59.554619 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:16:59.554595    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b46c2c15-92be-452b-9fd7-0ce038fbc9d9-model-cache\") pod \"s3-init-lq5rg\" (UID: \"b46c2c15-92be-452b-9fd7-0ce038fbc9d9\") " pod="kserve/s3-init-lq5rg"
Jul 28 10:16:59.562858 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:16:59.562834    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftgsz\" (UniqueName: \"kubernetes.io/projected/b46c2c15-92be-452b-9fd7-0ce038fbc9d9-kube-api-access-ftgsz\") pod \"s3-init-lq5rg\" (UID: \"b46c2c15-92be-452b-9fd7-0ce038fbc9d9\") " pod="kserve/s3-init-lq5rg"
Jul 28 10:16:59.682551 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:16:59.682512    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-lq5rg"
Jul 28 10:16:59.810367 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:16:59.810340    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-lq5rg"]
Jul 28 10:16:59.812359 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:16:59.812331    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb46c2c15_92be_452b_9fd7_0ce038fbc9d9.slice/crio-56e70b5f8fc62162d518ee823003be7582df1d2379eebb7b48e970df5b461074 WatchSource:0}: Error finding container 56e70b5f8fc62162d518ee823003be7582df1d2379eebb7b48e970df5b461074: Status 404 returned error can't find the container with id 56e70b5f8fc62162d518ee823003be7582df1d2379eebb7b48e970df5b461074
Jul 28 10:17:00.715368 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:17:00.715322    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-lq5rg" event={"ID":"b46c2c15-92be-452b-9fd7-0ce038fbc9d9","Type":"ContainerStarted","Data":"56e70b5f8fc62162d518ee823003be7582df1d2379eebb7b48e970df5b461074"}
Jul 28 10:17:03.730049 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:17:03.730010    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-lq5rg" event={"ID":"b46c2c15-92be-452b-9fd7-0ce038fbc9d9","Type":"ContainerStarted","Data":"493c8d8d5ee541a26698403cd178760203f75eab7d22271af42d15e9afd26632"}
Jul 28 10:17:04.735609 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:17:04.735575    2583 generic.go:356] "Generic (PLEG): container finished" podID="b46c2c15-92be-452b-9fd7-0ce038fbc9d9" containerID="493c8d8d5ee541a26698403cd178760203f75eab7d22271af42d15e9afd26632" exitCode=0
Jul 28 10:17:04.736007 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:17:04.735656    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-lq5rg" event={"ID":"b46c2c15-92be-452b-9fd7-0ce038fbc9d9","Type":"ContainerDied","Data":"493c8d8d5ee541a26698403cd178760203f75eab7d22271af42d15e9afd26632"}
Jul 28 10:17:09.761564 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:17:09.761525    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-lq5rg" event={"ID":"b46c2c15-92be-452b-9fd7-0ce038fbc9d9","Type":"ContainerStarted","Data":"cc20d08256996a7290ccabc680bacff223a980b82843357de0d5769259cfb7cc"}
Jul 28 10:17:09.780219 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:17:09.780157    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-lq5rg" podStartSLOduration=1.7229354510000001 podStartE2EDuration="10.780142327s" totalImagesPullingTime="7.408440371s" totalInitContainerRuntime="1.648766505s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:16:59 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:17:09.77877835 +0000 UTC m=+568.879284903" watchObservedRunningTime="2026-07-28 10:17:09.780142327 +0000 UTC m=+568.880648881"
Jul 28 10:17:24.476744 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:17:24.476694    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:18:11.967459 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:18:11.967363    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:18:14.043412 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:18:14.043378    2583 generic.go:356] "Generic (PLEG): container finished" podID="b46c2c15-92be-452b-9fd7-0ce038fbc9d9" containerID="cc20d08256996a7290ccabc680bacff223a980b82843357de0d5769259cfb7cc" exitCode=0
Jul 28 10:18:14.043741 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:18:14.043449    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-lq5rg" event={"ID":"b46c2c15-92be-452b-9fd7-0ce038fbc9d9","Type":"ContainerDied","Data":"cc20d08256996a7290ccabc680bacff223a980b82843357de0d5769259cfb7cc"}
Jul 28 10:18:15.172859 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:18:15.172830    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-lq5rg"
Jul 28 10:18:15.293019 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:18:15.292988    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ftgsz\" (UniqueName: \"kubernetes.io/projected/b46c2c15-92be-452b-9fd7-0ce038fbc9d9-kube-api-access-ftgsz\") pod \"b46c2c15-92be-452b-9fd7-0ce038fbc9d9\" (UID: \"b46c2c15-92be-452b-9fd7-0ce038fbc9d9\") "
Jul 28 10:18:15.293146 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:18:15.293048    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b46c2c15-92be-452b-9fd7-0ce038fbc9d9-model-cache\") pod \"b46c2c15-92be-452b-9fd7-0ce038fbc9d9\" (UID: \"b46c2c15-92be-452b-9fd7-0ce038fbc9d9\") "
Jul 28 10:18:15.295194 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:18:15.295156    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b46c2c15-92be-452b-9fd7-0ce038fbc9d9-kube-api-access-ftgsz" (OuterVolumeSpecName: "kube-api-access-ftgsz") pod "b46c2c15-92be-452b-9fd7-0ce038fbc9d9" (UID: "b46c2c15-92be-452b-9fd7-0ce038fbc9d9"). InnerVolumeSpecName "kube-api-access-ftgsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:18:15.311571 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:18:15.311545    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b46c2c15-92be-452b-9fd7-0ce038fbc9d9-model-cache" (OuterVolumeSpecName: "model-cache") pod "b46c2c15-92be-452b-9fd7-0ce038fbc9d9" (UID: "b46c2c15-92be-452b-9fd7-0ce038fbc9d9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:18:15.394190 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:18:15.394162    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b46c2c15-92be-452b-9fd7-0ce038fbc9d9-model-cache\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:18:15.394281 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:18:15.394210    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftgsz\" (UniqueName: \"kubernetes.io/projected/b46c2c15-92be-452b-9fd7-0ce038fbc9d9-kube-api-access-ftgsz\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:18:16.054341 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:18:16.054313    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-lq5rg"
Jul 28 10:18:16.054341 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:18:16.054324    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-lq5rg" event={"ID":"b46c2c15-92be-452b-9fd7-0ce038fbc9d9","Type":"ContainerDied","Data":"56e70b5f8fc62162d518ee823003be7582df1d2379eebb7b48e970df5b461074"}
Jul 28 10:18:16.054540 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:18:16.054367    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="56e70b5f8fc62162d518ee823003be7582df1d2379eebb7b48e970df5b461074"
Jul 28 10:18:16.466425 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:18:16.466392    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:18:16.774316 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:18:16.774243    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:18:28.022170 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:18:28.022132    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:18:28.097369 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:18:28.097341    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:18:28.104676 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:18:28.104644    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j"]
Jul 28 10:18:28.104992 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:18:28.104979    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b46c2c15-92be-452b-9fd7-0ce038fbc9d9" containerName="s3-init"
Jul 28 10:18:28.105039 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:18:28.104994    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b46c2c15-92be-452b-9fd7-0ce038fbc9d9" containerName="s3-init"
Jul 28 10:18:28.105039 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:18:28.105005    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b46c2c15-92be-452b-9fd7-0ce038fbc9d9" containerName="download-hf-model"
Jul 28 10:18:28.105039 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:18:28.105011    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b46c2c15-92be-452b-9fd7-0ce038fbc9d9" containerName="download-hf-model"
Jul 28 10:18:28.105145 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:18:28.105073    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b46c2c15-92be-452b-9fd7-0ce038fbc9d9" containerName="s3-init"
Jul 28 10:18:28.108241 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:18:28.108225    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j"
Jul 28 10:18:28.110438 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:18:28.110413    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 10:18:28.110548 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:18:28.110448    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 10:18:28.110548 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:18:28.110517    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-d7gr5\""
Jul 28 10:18:28.110915 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:18:28.110899    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 10:18:28.111209 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:18:28.111175    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 10:18:28.111331 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:18:28.111287    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-9d91d-predictor-serving-cert\""
Jul 28 10:18:28.111331 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:18:28.111294    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-9d91d-kube-rbac-proxy-sar-config\""
Jul 28 10:18:28.119466 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:18:28.119443    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j"]
Jul 28 10:18:28.193029 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:18:28.193002    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpg8b\" (UniqueName: \"kubernetes.io/projected/82a6a677-4ccf-4075-bf8b-5ce1d7f91586-kube-api-access-zpg8b\") pod \"isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j\" (UID: \"82a6a677-4ccf-4075-bf8b-5ce1d7f91586\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j"
Jul 28 10:18:28.193126 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:18:28.193033    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/82a6a677-4ccf-4075-bf8b-5ce1d7f91586-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j\" (UID: \"82a6a677-4ccf-4075-bf8b-5ce1d7f91586\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j"
Jul 28 10:18:28.193126 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:18:28.193055    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82a6a677-4ccf-4075-bf8b-5ce1d7f91586-proxy-tls\") pod \"isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j\" (UID: \"82a6a677-4ccf-4075-bf8b-5ce1d7f91586\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j"
Jul 28 10:18:28.193126 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:18:28.193096    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/82a6a677-4ccf-4075-bf8b-5ce1d7f91586-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j\" (UID: \"82a6a677-4ccf-4075-bf8b-5ce1d7f91586\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j"
Jul 28 10:18:28.193270 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:18:28.193149    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/82a6a677-4ccf-4075-bf8b-5ce1d7f91586-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j\" (UID: \"82a6a677-4ccf-4075-bf8b-5ce1d7f91586\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j"
Jul 28 10:18:28.294041 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:18:28.293979    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpg8b\" (UniqueName: \"kubernetes.io/projected/82a6a677-4ccf-4075-bf8b-5ce1d7f91586-kube-api-access-zpg8b\") pod \"isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j\" (UID: \"82a6a677-4ccf-4075-bf8b-5ce1d7f91586\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j"
Jul 28 10:18:28.294041 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:18:28.294016    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/82a6a677-4ccf-4075-bf8b-5ce1d7f91586-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j\" (UID: \"82a6a677-4ccf-4075-bf8b-5ce1d7f91586\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j"
Jul 28 10:18:28.294041 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:18:28.294037    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82a6a677-4ccf-4075-bf8b-5ce1d7f91586-proxy-tls\") pod \"isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j\" (UID: \"82a6a677-4ccf-4075-bf8b-5ce1d7f91586\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j"
Jul 28 10:18:28.294231 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:18:28.294062    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/82a6a677-4ccf-4075-bf8b-5ce1d7f91586-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j\" (UID: \"82a6a677-4ccf-4075-bf8b-5ce1d7f91586\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j"
Jul 28 10:18:28.294231 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:18:28.294085    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/82a6a677-4ccf-4075-bf8b-5ce1d7f91586-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j\" (UID: \"82a6a677-4ccf-4075-bf8b-5ce1d7f91586\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j"
Jul 28 10:18:28.294521 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:18:28.294496    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/82a6a677-4ccf-4075-bf8b-5ce1d7f91586-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j\" (UID: \"82a6a677-4ccf-4075-bf8b-5ce1d7f91586\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j"
Jul 28 10:18:28.294685 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:18:28.294664    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/82a6a677-4ccf-4075-bf8b-5ce1d7f91586-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j\" (UID: \"82a6a677-4ccf-4075-bf8b-5ce1d7f91586\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j"
Jul 28 10:18:28.294813 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:18:28.294793    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/82a6a677-4ccf-4075-bf8b-5ce1d7f91586-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j\" (UID: \"82a6a677-4ccf-4075-bf8b-5ce1d7f91586\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j"
Jul 28 10:18:28.296481 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:18:28.296454    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82a6a677-4ccf-4075-bf8b-5ce1d7f91586-proxy-tls\") pod \"isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j\" (UID: \"82a6a677-4ccf-4075-bf8b-5ce1d7f91586\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j"
Jul 28 10:18:28.303799 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:18:28.303776    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpg8b\" (UniqueName: \"kubernetes.io/projected/82a6a677-4ccf-4075-bf8b-5ce1d7f91586-kube-api-access-zpg8b\") pod \"isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j\" (UID: \"82a6a677-4ccf-4075-bf8b-5ce1d7f91586\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j"
Jul 28 10:18:28.419064 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:18:28.419041    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j"
Jul 28 10:18:28.549696 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:18:28.549642    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j"]
Jul 28 10:18:28.552131 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:18:28.552107    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82a6a677_4ccf_4075_bf8b_5ce1d7f91586.slice/crio-300bf3cd1a464a8eac3927ff56d414669d429a205584f4db409d79c1ad7954c2 WatchSource:0}: Error finding container 300bf3cd1a464a8eac3927ff56d414669d429a205584f4db409d79c1ad7954c2: Status 404 returned error can't find the container with id 300bf3cd1a464a8eac3927ff56d414669d429a205584f4db409d79c1ad7954c2
Jul 28 10:18:29.111812 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:18:29.111781    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j" event={"ID":"82a6a677-4ccf-4075-bf8b-5ce1d7f91586","Type":"ContainerStarted","Data":"5756f402df5c8aedcf0b86c269c957766cb5dc59187ca76aff29a5389bed2234"}
Jul 28 10:18:29.112196 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:18:29.111842    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j" event={"ID":"82a6a677-4ccf-4075-bf8b-5ce1d7f91586","Type":"ContainerStarted","Data":"300bf3cd1a464a8eac3927ff56d414669d429a205584f4db409d79c1ad7954c2"}
Jul 28 10:18:33.131079 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:18:33.131042    2583 generic.go:356] "Generic (PLEG): container finished" podID="82a6a677-4ccf-4075-bf8b-5ce1d7f91586" containerID="5756f402df5c8aedcf0b86c269c957766cb5dc59187ca76aff29a5389bed2234" exitCode=0
Jul 28 10:18:33.131489 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:18:33.131120    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j" event={"ID":"82a6a677-4ccf-4075-bf8b-5ce1d7f91586","Type":"ContainerDied","Data":"5756f402df5c8aedcf0b86c269c957766cb5dc59187ca76aff29a5389bed2234"}
Jul 28 10:18:46.261289 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:18:46.261270    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 10:18:47.208421 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:18:47.208381    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j" event={"ID":"82a6a677-4ccf-4075-bf8b-5ce1d7f91586","Type":"ContainerStarted","Data":"209815e35ea920b486c3014ed0af7244662c395a2378ea0e93619c8b4f1ec290"}
Jul 28 10:18:49.221282 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:18:49.221243    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j" event={"ID":"82a6a677-4ccf-4075-bf8b-5ce1d7f91586","Type":"ContainerStarted","Data":"7b15c8b237aa16000d3b3042fae8752ef888ceaeee849919e5f89b45c73fe03e"}
Jul 28 10:18:51.235157 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:18:51.235072    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j" event={"ID":"82a6a677-4ccf-4075-bf8b-5ce1d7f91586","Type":"ContainerStarted","Data":"e0a530cd51414ccc0ef319781d2d7c3e071193f31f9dac412dd47a6777744d5a"}
Jul 28 10:18:51.235628 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:18:51.235201    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j"
Jul 28 10:18:52.240430 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:18:52.240396    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j"
Jul 28 10:18:52.240786 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:18:52.240486    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j"
Jul 28 10:18:52.241792 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:18:52.241747    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j" podUID="82a6a677-4ccf-4075-bf8b-5ce1d7f91586" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 10:18:52.242350 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:18:52.242329    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j" podUID="82a6a677-4ccf-4075-bf8b-5ce1d7f91586" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 10:18:53.244993 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:18:53.244950    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j" podUID="82a6a677-4ccf-4075-bf8b-5ce1d7f91586" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 10:18:53.245447 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:18:53.245348    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j" podUID="82a6a677-4ccf-4075-bf8b-5ce1d7f91586" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 10:18:53.248542 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:18:53.248524    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j"
Jul 28 10:18:53.270609 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:18:53.270568    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j" podStartSLOduration=3.702230555 podStartE2EDuration="25.270555639s" totalImagesPullingTime="17.704460662s" totalInitContainerRuntime="3.863864422s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:18:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:18:51.256682332 +0000 UTC m=+670.357188908" watchObservedRunningTime="2026-07-28 10:18:53.270555639 +0000 UTC m=+672.371062193"
Jul 28 10:18:54.248930 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:18:54.248884    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j" podUID="82a6a677-4ccf-4075-bf8b-5ce1d7f91586" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 10:18:54.249357 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:18:54.249239    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j" podUID="82a6a677-4ccf-4075-bf8b-5ce1d7f91586" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 10:19:01.584806 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:19:01.584775    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:19:04.249517 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:19:04.249473    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j" podUID="82a6a677-4ccf-4075-bf8b-5ce1d7f91586" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 10:19:04.250556 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:19:04.250516    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j" podUID="82a6a677-4ccf-4075-bf8b-5ce1d7f91586" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 10:19:14.249587 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:19:14.249538    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j" podUID="82a6a677-4ccf-4075-bf8b-5ce1d7f91586" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 10:19:14.250092 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:19:14.250066    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j" podUID="82a6a677-4ccf-4075-bf8b-5ce1d7f91586" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 10:19:23.791465 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:19:23.791432    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:19:24.248975 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:19:24.248935    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j" podUID="82a6a677-4ccf-4075-bf8b-5ce1d7f91586" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 10:19:24.249400 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:19:24.249371    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j" podUID="82a6a677-4ccf-4075-bf8b-5ce1d7f91586" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 10:19:34.249105 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:19:34.249011    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j" podUID="82a6a677-4ccf-4075-bf8b-5ce1d7f91586" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 10:19:34.249524 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:19:34.249433    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j" podUID="82a6a677-4ccf-4075-bf8b-5ce1d7f91586" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 10:19:44.249169 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:19:44.249120    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j" podUID="82a6a677-4ccf-4075-bf8b-5ce1d7f91586" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 10:19:44.249620 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:19:44.249556    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j" podUID="82a6a677-4ccf-4075-bf8b-5ce1d7f91586" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 10:19:54.249601 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:19:54.249553    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j" podUID="82a6a677-4ccf-4075-bf8b-5ce1d7f91586" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 10:19:54.250136 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:19:54.249965    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j" podUID="82a6a677-4ccf-4075-bf8b-5ce1d7f91586" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 10:20:04.249439 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:04.249381    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j"
Jul 28 10:20:04.250000 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:04.249741    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j"
Jul 28 10:20:13.245875 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:13.245844    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:20:13.299533 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:13.299496    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j"]
Jul 28 10:20:13.300093 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:13.300046    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j" podUID="82a6a677-4ccf-4075-bf8b-5ce1d7f91586" containerName="kserve-container" containerID="cri-o://209815e35ea920b486c3014ed0af7244662c395a2378ea0e93619c8b4f1ec290" gracePeriod=30
Jul 28 10:20:13.300301 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:13.300114    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j" podUID="82a6a677-4ccf-4075-bf8b-5ce1d7f91586" containerName="agent" containerID="cri-o://e0a530cd51414ccc0ef319781d2d7c3e071193f31f9dac412dd47a6777744d5a" gracePeriod=30
Jul 28 10:20:13.300423 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:13.300078    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j" podUID="82a6a677-4ccf-4075-bf8b-5ce1d7f91586" containerName="kube-rbac-proxy" containerID="cri-o://7b15c8b237aa16000d3b3042fae8752ef888ceaeee849919e5f89b45c73fe03e" gracePeriod=30
Jul 28 10:20:13.309039 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:13.308998    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m"]
Jul 28 10:20:13.313330 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:13.313311    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m"
Jul 28 10:20:13.315474 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:13.315454    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-74e2d-predictor-serving-cert\""
Jul 28 10:20:13.315574 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:13.315472    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-74e2d-kube-rbac-proxy-sar-config\""
Jul 28 10:20:13.331281 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:13.331259    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m"]
Jul 28 10:20:13.331903 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:13.331885    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:20:13.370111 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:13.370083    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp"]
Jul 28 10:20:13.373232 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:13.373216    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp"
Jul 28 10:20:13.375136 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:13.375111    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-74e2d-predictor-serving-cert\""
Jul 28 10:20:13.375268 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:13.375253    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-74e2d-kube-rbac-proxy-sar-config\""
Jul 28 10:20:13.383947 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:13.383926    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp"]
Jul 28 10:20:13.388303 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:13.388283    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ecf844c8-0bcc-4e5a-8d75-301f9560c753-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m\" (UID: \"ecf844c8-0bcc-4e5a-8d75-301f9560c753\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m"
Jul 28 10:20:13.388401 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:13.388336    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ecf844c8-0bcc-4e5a-8d75-301f9560c753-cabundle-cert\") pod \"isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m\" (UID: \"ecf844c8-0bcc-4e5a-8d75-301f9560c753\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m"
Jul 28 10:20:13.388458 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:13.388422    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ecf844c8-0bcc-4e5a-8d75-301f9560c753-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m\" (UID: \"ecf844c8-0bcc-4e5a-8d75-301f9560c753\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m"
Jul 28 10:20:13.388501 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:13.388472    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw8t2\" (UniqueName: \"kubernetes.io/projected/ecf844c8-0bcc-4e5a-8d75-301f9560c753-kube-api-access-zw8t2\") pod \"isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m\" (UID: \"ecf844c8-0bcc-4e5a-8d75-301f9560c753\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m"
Jul 28 10:20:13.388569 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:13.388549    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ecf844c8-0bcc-4e5a-8d75-301f9560c753-proxy-tls\") pod \"isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m\" (UID: \"ecf844c8-0bcc-4e5a-8d75-301f9560c753\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m"
Jul 28 10:20:13.396357 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:13.396337    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:20:13.489366 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:13.489336    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/285d0b61-573d-4069-82fd-f1a248dbb07f-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp\" (UID: \"285d0b61-573d-4069-82fd-f1a248dbb07f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp"
Jul 28 10:20:13.489498 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:13.489394    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/285d0b61-573d-4069-82fd-f1a248dbb07f-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp\" (UID: \"285d0b61-573d-4069-82fd-f1a248dbb07f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp"
Jul 28 10:20:13.489498 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:13.489432    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ecf844c8-0bcc-4e5a-8d75-301f9560c753-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m\" (UID: \"ecf844c8-0bcc-4e5a-8d75-301f9560c753\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m"
Jul 28 10:20:13.489498 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:13.489458    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/285d0b61-573d-4069-82fd-f1a248dbb07f-cabundle-cert\") pod \"isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp\" (UID: \"285d0b61-573d-4069-82fd-f1a248dbb07f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp"
Jul 28 10:20:13.489498 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:13.489492    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ecf844c8-0bcc-4e5a-8d75-301f9560c753-cabundle-cert\") pod \"isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m\" (UID: \"ecf844c8-0bcc-4e5a-8d75-301f9560c753\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m"
Jul 28 10:20:13.489700 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:13.489574    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk5th\" (UniqueName: \"kubernetes.io/projected/285d0b61-573d-4069-82fd-f1a248dbb07f-kube-api-access-nk5th\") pod \"isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp\" (UID: \"285d0b61-573d-4069-82fd-f1a248dbb07f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp"
Jul 28 10:20:13.489700 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:13.489655    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ecf844c8-0bcc-4e5a-8d75-301f9560c753-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m\" (UID: \"ecf844c8-0bcc-4e5a-8d75-301f9560c753\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m"
Jul 28 10:20:13.489813 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:13.489766    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw8t2\" (UniqueName: \"kubernetes.io/projected/ecf844c8-0bcc-4e5a-8d75-301f9560c753-kube-api-access-zw8t2\") pod \"isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m\" (UID: \"ecf844c8-0bcc-4e5a-8d75-301f9560c753\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m"
Jul 28 10:20:13.489813 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:13.489794    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/285d0b61-573d-4069-82fd-f1a248dbb07f-proxy-tls\") pod \"isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp\" (UID: \"285d0b61-573d-4069-82fd-f1a248dbb07f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp"
Jul 28 10:20:13.489917 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:13.489840    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ecf844c8-0bcc-4e5a-8d75-301f9560c753-proxy-tls\") pod \"isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m\" (UID: \"ecf844c8-0bcc-4e5a-8d75-301f9560c753\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m"
Jul 28 10:20:13.489917 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:13.489842    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ecf844c8-0bcc-4e5a-8d75-301f9560c753-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m\" (UID: \"ecf844c8-0bcc-4e5a-8d75-301f9560c753\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m"
Jul 28 10:20:13.490154 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:20:13.490125    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-raw-74e2d-predictor-serving-cert: secret "isvc-sklearn-graph-raw-74e2d-predictor-serving-cert" not found
Jul 28 10:20:13.490266 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:20:13.490226    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecf844c8-0bcc-4e5a-8d75-301f9560c753-proxy-tls podName:ecf844c8-0bcc-4e5a-8d75-301f9560c753 nodeName:}" failed. No retries permitted until 2026-07-28 10:20:13.990207498 +0000 UTC m=+753.090714037 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ecf844c8-0bcc-4e5a-8d75-301f9560c753-proxy-tls") pod "isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m" (UID: "ecf844c8-0bcc-4e5a-8d75-301f9560c753") : secret "isvc-sklearn-graph-raw-74e2d-predictor-serving-cert" not found
Jul 28 10:20:13.490345 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:13.490304    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ecf844c8-0bcc-4e5a-8d75-301f9560c753-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m\" (UID: \"ecf844c8-0bcc-4e5a-8d75-301f9560c753\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m"
Jul 28 10:20:13.490695 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:13.490673    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ecf844c8-0bcc-4e5a-8d75-301f9560c753-cabundle-cert\") pod \"isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m\" (UID: \"ecf844c8-0bcc-4e5a-8d75-301f9560c753\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m"
Jul 28 10:20:13.497731 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:13.497680    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw8t2\" (UniqueName: \"kubernetes.io/projected/ecf844c8-0bcc-4e5a-8d75-301f9560c753-kube-api-access-zw8t2\") pod \"isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m\" (UID: \"ecf844c8-0bcc-4e5a-8d75-301f9560c753\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m"
Jul 28 10:20:13.590775 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:13.590751    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/285d0b61-573d-4069-82fd-f1a248dbb07f-proxy-tls\") pod \"isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp\" (UID: \"285d0b61-573d-4069-82fd-f1a248dbb07f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp"
Jul 28 10:20:13.590886 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:13.590821    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/285d0b61-573d-4069-82fd-f1a248dbb07f-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp\" (UID: \"285d0b61-573d-4069-82fd-f1a248dbb07f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp"
Jul 28 10:20:13.590886 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:13.590862    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/285d0b61-573d-4069-82fd-f1a248dbb07f-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp\" (UID: \"285d0b61-573d-4069-82fd-f1a248dbb07f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp"
Jul 28 10:20:13.590886 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:20:13.590874    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-raw-74e2d-predictor-serving-cert: secret "isvc-xgboost-graph-raw-74e2d-predictor-serving-cert" not found
Jul 28 10:20:13.591053 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:13.590891    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/285d0b61-573d-4069-82fd-f1a248dbb07f-cabundle-cert\") pod \"isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp\" (UID: \"285d0b61-573d-4069-82fd-f1a248dbb07f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp"
Jul 28 10:20:13.591053 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:20:13.590943    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/285d0b61-573d-4069-82fd-f1a248dbb07f-proxy-tls podName:285d0b61-573d-4069-82fd-f1a248dbb07f nodeName:}" failed. No retries permitted until 2026-07-28 10:20:14.090924571 +0000 UTC m=+753.191431105 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/285d0b61-573d-4069-82fd-f1a248dbb07f-proxy-tls") pod "isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp" (UID: "285d0b61-573d-4069-82fd-f1a248dbb07f") : secret "isvc-xgboost-graph-raw-74e2d-predictor-serving-cert" not found
Jul 28 10:20:13.591053 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:13.590964    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk5th\" (UniqueName: \"kubernetes.io/projected/285d0b61-573d-4069-82fd-f1a248dbb07f-kube-api-access-nk5th\") pod \"isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp\" (UID: \"285d0b61-573d-4069-82fd-f1a248dbb07f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp"
Jul 28 10:20:13.591515 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:13.591486    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/285d0b61-573d-4069-82fd-f1a248dbb07f-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp\" (UID: \"285d0b61-573d-4069-82fd-f1a248dbb07f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp"
Jul 28 10:20:13.591596 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:13.591554    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/285d0b61-573d-4069-82fd-f1a248dbb07f-cabundle-cert\") pod \"isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp\" (UID: \"285d0b61-573d-4069-82fd-f1a248dbb07f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp"
Jul 28 10:20:13.591848 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:13.591826    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/285d0b61-573d-4069-82fd-f1a248dbb07f-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp\" (UID: \"285d0b61-573d-4069-82fd-f1a248dbb07f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp"
Jul 28 10:20:13.594957 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:13.594934    2583 generic.go:356] "Generic (PLEG): container finished" podID="82a6a677-4ccf-4075-bf8b-5ce1d7f91586" containerID="7b15c8b237aa16000d3b3042fae8752ef888ceaeee849919e5f89b45c73fe03e" exitCode=2
Jul 28 10:20:13.595035 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:13.595012    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j" event={"ID":"82a6a677-4ccf-4075-bf8b-5ce1d7f91586","Type":"ContainerDied","Data":"7b15c8b237aa16000d3b3042fae8752ef888ceaeee849919e5f89b45c73fe03e"}
Jul 28 10:20:13.599030 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:13.599011    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk5th\" (UniqueName: \"kubernetes.io/projected/285d0b61-573d-4069-82fd-f1a248dbb07f-kube-api-access-nk5th\") pod \"isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp\" (UID: \"285d0b61-573d-4069-82fd-f1a248dbb07f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp"
Jul 28 10:20:13.837021 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:13.836953    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:20:13.936603 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:13.936578    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:20:13.994993 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:13.994967    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ecf844c8-0bcc-4e5a-8d75-301f9560c753-proxy-tls\") pod \"isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m\" (UID: \"ecf844c8-0bcc-4e5a-8d75-301f9560c753\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m"
Jul 28 10:20:13.997445 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:13.997428    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ecf844c8-0bcc-4e5a-8d75-301f9560c753-proxy-tls\") pod \"isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m\" (UID: \"ecf844c8-0bcc-4e5a-8d75-301f9560c753\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m"
Jul 28 10:20:14.096103 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:14.096050    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/285d0b61-573d-4069-82fd-f1a248dbb07f-proxy-tls\") pod \"isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp\" (UID: \"285d0b61-573d-4069-82fd-f1a248dbb07f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp"
Jul 28 10:20:14.098263 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:14.098245    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/285d0b61-573d-4069-82fd-f1a248dbb07f-proxy-tls\") pod \"isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp\" (UID: \"285d0b61-573d-4069-82fd-f1a248dbb07f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp"
Jul 28 10:20:14.228754 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:14.228725    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m"
Jul 28 10:20:14.249757 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:14.249724    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j" podUID="82a6a677-4ccf-4075-bf8b-5ce1d7f91586" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 10:20:14.250103 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:14.250078    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j" podUID="82a6a677-4ccf-4075-bf8b-5ce1d7f91586" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 10:20:14.284088 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:14.284068    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp"
Jul 28 10:20:14.366166 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:14.366145    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m"]
Jul 28 10:20:14.369554 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:20:14.369525    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecf844c8_0bcc_4e5a_8d75_301f9560c753.slice/crio-60ad422c7427c10924260fd5aa9629bd1790ed1d01e01910f4959c9734c839a2 WatchSource:0}: Error finding container 60ad422c7427c10924260fd5aa9629bd1790ed1d01e01910f4959c9734c839a2: Status 404 returned error can't find the container with id 60ad422c7427c10924260fd5aa9629bd1790ed1d01e01910f4959c9734c839a2
Jul 28 10:20:14.428331 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:14.428310    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp"]
Jul 28 10:20:14.430302 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:20:14.430272    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod285d0b61_573d_4069_82fd_f1a248dbb07f.slice/crio-0de34e18dc5f5cd744d16be7a58a38275db5a8c07591ff893decfc40ef5055b3 WatchSource:0}: Error finding container 0de34e18dc5f5cd744d16be7a58a38275db5a8c07591ff893decfc40ef5055b3: Status 404 returned error can't find the container with id 0de34e18dc5f5cd744d16be7a58a38275db5a8c07591ff893decfc40ef5055b3
Jul 28 10:20:14.601447 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:14.601398    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m" event={"ID":"ecf844c8-0bcc-4e5a-8d75-301f9560c753","Type":"ContainerStarted","Data":"64d7e07f34a87c2fc8a0626318fc7872c4335866e445c07f3c6d5c075f427664"}
Jul 28 10:20:14.601656 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:14.601484    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m" event={"ID":"ecf844c8-0bcc-4e5a-8d75-301f9560c753","Type":"ContainerStarted","Data":"60ad422c7427c10924260fd5aa9629bd1790ed1d01e01910f4959c9734c839a2"}
Jul 28 10:20:14.603093 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:14.603066    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp" event={"ID":"285d0b61-573d-4069-82fd-f1a248dbb07f","Type":"ContainerStarted","Data":"db07a8689b8da2639161cf393883ae59fa21c14f9609d44c1339bd9e30862a30"}
Jul 28 10:20:14.603231 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:14.603112    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp" event={"ID":"285d0b61-573d-4069-82fd-f1a248dbb07f","Type":"ContainerStarted","Data":"0de34e18dc5f5cd744d16be7a58a38275db5a8c07591ff893decfc40ef5055b3"}
Jul 28 10:20:17.618411 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:17.618341    2583 generic.go:356] "Generic (PLEG): container finished" podID="82a6a677-4ccf-4075-bf8b-5ce1d7f91586" containerID="209815e35ea920b486c3014ed0af7244662c395a2378ea0e93619c8b4f1ec290" exitCode=0
Jul 28 10:20:17.618411 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:17.618370    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j" event={"ID":"82a6a677-4ccf-4075-bf8b-5ce1d7f91586","Type":"ContainerDied","Data":"209815e35ea920b486c3014ed0af7244662c395a2378ea0e93619c8b4f1ec290"}
Jul 28 10:20:18.245347 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:18.245311    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j" podUID="82a6a677-4ccf-4075-bf8b-5ce1d7f91586" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Jul 28 10:20:18.630443 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:18.630411    2583 generic.go:356] "Generic (PLEG): container finished" podID="ecf844c8-0bcc-4e5a-8d75-301f9560c753" containerID="64d7e07f34a87c2fc8a0626318fc7872c4335866e445c07f3c6d5c075f427664" exitCode=0
Jul 28 10:20:18.630850 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:18.630493    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m" event={"ID":"ecf844c8-0bcc-4e5a-8d75-301f9560c753","Type":"ContainerDied","Data":"64d7e07f34a87c2fc8a0626318fc7872c4335866e445c07f3c6d5c075f427664"}
Jul 28 10:20:18.632294 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:18.632267    2583 generic.go:356] "Generic (PLEG): container finished" podID="285d0b61-573d-4069-82fd-f1a248dbb07f" containerID="db07a8689b8da2639161cf393883ae59fa21c14f9609d44c1339bd9e30862a30" exitCode=0
Jul 28 10:20:18.632386 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:18.632327    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp" event={"ID":"285d0b61-573d-4069-82fd-f1a248dbb07f","Type":"ContainerDied","Data":"db07a8689b8da2639161cf393883ae59fa21c14f9609d44c1339bd9e30862a30"}
Jul 28 10:20:19.641688 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:19.641642    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m" event={"ID":"ecf844c8-0bcc-4e5a-8d75-301f9560c753","Type":"ContainerStarted","Data":"7642a0c012c7e5f04913b477af40ce75e168b8475a0b63977f372836f3f1d5f0"}
Jul 28 10:20:19.642111 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:19.641728    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m" event={"ID":"ecf844c8-0bcc-4e5a-8d75-301f9560c753","Type":"ContainerStarted","Data":"5ff08d35b66c6d1b9b15e748e1d7c050d608fcb23aa37d753951fdf70794e6fa"}
Jul 28 10:20:19.642213 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:19.642171    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m"
Jul 28 10:20:19.642273 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:19.642242    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m"
Jul 28 10:20:19.643837 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:19.643803    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m" podUID="ecf844c8-0bcc-4e5a-8d75-301f9560c753" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 28 10:20:20.648597 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:20.648426    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m" podUID="ecf844c8-0bcc-4e5a-8d75-301f9560c753" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 28 10:20:23.245629 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:23.245590    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j" podUID="82a6a677-4ccf-4075-bf8b-5ce1d7f91586" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Jul 28 10:20:24.249698 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:24.249637    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j" podUID="82a6a677-4ccf-4075-bf8b-5ce1d7f91586" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 10:20:24.250150 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:24.249992    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j" podUID="82a6a677-4ccf-4075-bf8b-5ce1d7f91586" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 10:20:25.653749 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:25.653719    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m"
Jul 28 10:20:25.654256 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:25.654220    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m" podUID="ecf844c8-0bcc-4e5a-8d75-301f9560c753" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 28 10:20:25.673170 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:25.673117    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m" podStartSLOduration=9.038681705 podStartE2EDuration="12.673098548s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.634416843s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:20:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:20:19.661912303 +0000 UTC m=+758.762418853" watchObservedRunningTime="2026-07-28 10:20:25.673098548 +0000 UTC m=+764.773605102"
Jul 28 10:20:28.245967 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:28.245928    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j" podUID="82a6a677-4ccf-4075-bf8b-5ce1d7f91586" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Jul 28 10:20:28.246391 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:28.246080    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j"
Jul 28 10:20:33.245961 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:33.245899    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j" podUID="82a6a677-4ccf-4075-bf8b-5ce1d7f91586" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Jul 28 10:20:34.249439 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:34.249387    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j" podUID="82a6a677-4ccf-4075-bf8b-5ce1d7f91586" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 10:20:34.249891 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:34.249559    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j"
Jul 28 10:20:34.249891 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:34.249796    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j" podUID="82a6a677-4ccf-4075-bf8b-5ce1d7f91586" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 10:20:34.250002 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:34.249909    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j"
Jul 28 10:20:35.655087 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:35.655045    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m" podUID="ecf844c8-0bcc-4e5a-8d75-301f9560c753" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 28 10:20:36.732102 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:36.732063    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp" event={"ID":"285d0b61-573d-4069-82fd-f1a248dbb07f","Type":"ContainerStarted","Data":"2a2f93c9e79cc8bc530de384c50a2486f51d4cea2942a3b52b85e6989a7994c5"}
Jul 28 10:20:36.732523 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:36.732130    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp" event={"ID":"285d0b61-573d-4069-82fd-f1a248dbb07f","Type":"ContainerStarted","Data":"27c1fcb73b14ec1f454621cca50be756972a4b23ac3d56b5129bd2731782362e"}
Jul 28 10:20:36.732523 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:36.732456    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp"
Jul 28 10:20:36.732523 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:36.732509    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp"
Jul 28 10:20:36.733538 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:36.733500    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp" podUID="285d0b61-573d-4069-82fd-f1a248dbb07f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 28 10:20:36.752401 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:36.752361    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp" podStartSLOduration=2.910540269 podStartE2EDuration="23.752348094s" totalImagesPullingTime="17.222608267s" totalInitContainerRuntime="3.619199558s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:20:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:20:36.750861047 +0000 UTC m=+775.851367601" watchObservedRunningTime="2026-07-28 10:20:36.752348094 +0000 UTC m=+775.852854647"
Jul 28 10:20:37.737090 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:37.737049    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp" podUID="285d0b61-573d-4069-82fd-f1a248dbb07f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 28 10:20:38.245650 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:38.245609    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j" podUID="82a6a677-4ccf-4075-bf8b-5ce1d7f91586" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Jul 28 10:20:42.741673 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:42.741643    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp"
Jul 28 10:20:42.742161 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:42.742129    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp" podUID="285d0b61-573d-4069-82fd-f1a248dbb07f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 28 10:20:43.245480 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:43.245443    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j" podUID="82a6a677-4ccf-4075-bf8b-5ce1d7f91586" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Jul 28 10:20:43.766752 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:43.766719    2583 generic.go:356] "Generic (PLEG): container finished" podID="82a6a677-4ccf-4075-bf8b-5ce1d7f91586" containerID="e0a530cd51414ccc0ef319781d2d7c3e071193f31f9dac412dd47a6777744d5a" exitCode=0
Jul 28 10:20:43.767094 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:43.766786    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j" event={"ID":"82a6a677-4ccf-4075-bf8b-5ce1d7f91586","Type":"ContainerDied","Data":"e0a530cd51414ccc0ef319781d2d7c3e071193f31f9dac412dd47a6777744d5a"}
Jul 28 10:20:43.971166 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:43.971143    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j"
Jul 28 10:20:44.039975 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:44.039948    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/82a6a677-4ccf-4075-bf8b-5ce1d7f91586-kserve-provision-location\") pod \"82a6a677-4ccf-4075-bf8b-5ce1d7f91586\" (UID: \"82a6a677-4ccf-4075-bf8b-5ce1d7f91586\") "
Jul 28 10:20:44.040101 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:44.039996    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zpg8b\" (UniqueName: \"kubernetes.io/projected/82a6a677-4ccf-4075-bf8b-5ce1d7f91586-kube-api-access-zpg8b\") pod \"82a6a677-4ccf-4075-bf8b-5ce1d7f91586\" (UID: \"82a6a677-4ccf-4075-bf8b-5ce1d7f91586\") "
Jul 28 10:20:44.040101 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:44.040046    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/82a6a677-4ccf-4075-bf8b-5ce1d7f91586-kube-rbac-proxy-sar-config\") pod \"82a6a677-4ccf-4075-bf8b-5ce1d7f91586\" (UID: \"82a6a677-4ccf-4075-bf8b-5ce1d7f91586\") "
Jul 28 10:20:44.040101 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:44.040067    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82a6a677-4ccf-4075-bf8b-5ce1d7f91586-proxy-tls\") pod \"82a6a677-4ccf-4075-bf8b-5ce1d7f91586\" (UID: \"82a6a677-4ccf-4075-bf8b-5ce1d7f91586\") "
Jul 28 10:20:44.040101 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:44.040093    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/82a6a677-4ccf-4075-bf8b-5ce1d7f91586-cabundle-cert\") pod \"82a6a677-4ccf-4075-bf8b-5ce1d7f91586\" (UID: \"82a6a677-4ccf-4075-bf8b-5ce1d7f91586\") "
Jul 28 10:20:44.040325 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:44.040254    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82a6a677-4ccf-4075-bf8b-5ce1d7f91586-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "82a6a677-4ccf-4075-bf8b-5ce1d7f91586" (UID: "82a6a677-4ccf-4075-bf8b-5ce1d7f91586"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:20:44.040381 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:44.040334    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/82a6a677-4ccf-4075-bf8b-5ce1d7f91586-kserve-provision-location\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:20:44.040467 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:44.040449    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82a6a677-4ccf-4075-bf8b-5ce1d7f91586-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "82a6a677-4ccf-4075-bf8b-5ce1d7f91586" (UID: "82a6a677-4ccf-4075-bf8b-5ce1d7f91586"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:20:44.040534 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:44.040508    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82a6a677-4ccf-4075-bf8b-5ce1d7f91586-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "82a6a677-4ccf-4075-bf8b-5ce1d7f91586" (UID: "82a6a677-4ccf-4075-bf8b-5ce1d7f91586"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:20:44.042222 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:44.042201    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82a6a677-4ccf-4075-bf8b-5ce1d7f91586-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "82a6a677-4ccf-4075-bf8b-5ce1d7f91586" (UID: "82a6a677-4ccf-4075-bf8b-5ce1d7f91586"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:20:44.042305 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:44.042201    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82a6a677-4ccf-4075-bf8b-5ce1d7f91586-kube-api-access-zpg8b" (OuterVolumeSpecName: "kube-api-access-zpg8b") pod "82a6a677-4ccf-4075-bf8b-5ce1d7f91586" (UID: "82a6a677-4ccf-4075-bf8b-5ce1d7f91586"). InnerVolumeSpecName "kube-api-access-zpg8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:20:44.141317 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:44.141261    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/82a6a677-4ccf-4075-bf8b-5ce1d7f91586-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:20:44.141317 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:44.141287    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82a6a677-4ccf-4075-bf8b-5ce1d7f91586-proxy-tls\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:20:44.141317 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:44.141300    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/82a6a677-4ccf-4075-bf8b-5ce1d7f91586-cabundle-cert\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:20:44.141317 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:44.141312    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpg8b\" (UniqueName: \"kubernetes.io/projected/82a6a677-4ccf-4075-bf8b-5ce1d7f91586-kube-api-access-zpg8b\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:20:44.773764 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:44.773735    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j"
Jul 28 10:20:44.774157 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:44.773758    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j" event={"ID":"82a6a677-4ccf-4075-bf8b-5ce1d7f91586","Type":"ContainerDied","Data":"300bf3cd1a464a8eac3927ff56d414669d429a205584f4db409d79c1ad7954c2"}
Jul 28 10:20:44.774157 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:44.773834    2583 scope.go:117] "RemoveContainer" containerID="e0a530cd51414ccc0ef319781d2d7c3e071193f31f9dac412dd47a6777744d5a"
Jul 28 10:20:44.782908 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:44.782892    2583 scope.go:117] "RemoveContainer" containerID="7b15c8b237aa16000d3b3042fae8752ef888ceaeee849919e5f89b45c73fe03e"
Jul 28 10:20:44.790225 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:44.790208    2583 scope.go:117] "RemoveContainer" containerID="209815e35ea920b486c3014ed0af7244662c395a2378ea0e93619c8b4f1ec290"
Jul 28 10:20:44.797205 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:44.797170    2583 scope.go:117] "RemoveContainer" containerID="5756f402df5c8aedcf0b86c269c957766cb5dc59187ca76aff29a5389bed2234"
Jul 28 10:20:44.797532 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:44.797511    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j"]
Jul 28 10:20:44.804022 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:44.804002    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9d91d-predictor-78dd555d7c-qnf9j"]
Jul 28 10:20:45.439142 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:45.439108    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82a6a677-4ccf-4075-bf8b-5ce1d7f91586" path="/var/lib/kubelet/pods/82a6a677-4ccf-4075-bf8b-5ce1d7f91586/volumes"
Jul 28 10:20:45.654481 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:45.654440    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m" podUID="ecf844c8-0bcc-4e5a-8d75-301f9560c753" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 28 10:20:52.742330 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:52.742287    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp" podUID="285d0b61-573d-4069-82fd-f1a248dbb07f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 28 10:20:55.654789 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:20:55.654752    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m" podUID="ecf844c8-0bcc-4e5a-8d75-301f9560c753" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 28 10:21:02.743003 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:21:02.742965    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp" podUID="285d0b61-573d-4069-82fd-f1a248dbb07f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 28 10:21:05.655019 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:21:05.654983    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m" podUID="ecf844c8-0bcc-4e5a-8d75-301f9560c753" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 28 10:21:12.742351 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:21:12.742312    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp" podUID="285d0b61-573d-4069-82fd-f1a248dbb07f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 28 10:21:15.654276 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:21:15.654240    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m" podUID="ecf844c8-0bcc-4e5a-8d75-301f9560c753" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 28 10:21:22.742673 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:21:22.742635    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp" podUID="285d0b61-573d-4069-82fd-f1a248dbb07f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 28 10:21:25.654824 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:21:25.654780    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m" podUID="ecf844c8-0bcc-4e5a-8d75-301f9560c753" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 28 10:21:32.743000 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:21:32.742969    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp"
Jul 28 10:21:35.654796 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:21:35.654763    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m"
Jul 28 10:21:49.786653 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:21:49.786618    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:21:53.411208 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:21:53.411148    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:21:53.487363 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:21:53.487328    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:22:03.418292 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:03.418256    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:22:03.614649 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:03.614615    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m"]
Jul 28 10:22:03.614904 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:03.614880    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m" podUID="ecf844c8-0bcc-4e5a-8d75-301f9560c753" containerName="kserve-container" containerID="cri-o://5ff08d35b66c6d1b9b15e748e1d7c050d608fcb23aa37d753951fdf70794e6fa" gracePeriod=30
Jul 28 10:22:03.615004 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:03.614988    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:22:03.615120 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:03.615096    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m" podUID="ecf844c8-0bcc-4e5a-8d75-301f9560c753" containerName="kube-rbac-proxy" containerID="cri-o://7642a0c012c7e5f04913b477af40ce75e168b8475a0b63977f372836f3f1d5f0" gracePeriod=30
Jul 28 10:22:03.669632 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:03.669569    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:22:03.672848 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:03.672828    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq"]
Jul 28 10:22:03.673144 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:03.673132    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82a6a677-4ccf-4075-bf8b-5ce1d7f91586" containerName="kserve-container"
Jul 28 10:22:03.673198 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:03.673146    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="82a6a677-4ccf-4075-bf8b-5ce1d7f91586" containerName="kserve-container"
Jul 28 10:22:03.673198 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:03.673160    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82a6a677-4ccf-4075-bf8b-5ce1d7f91586" containerName="agent"
Jul 28 10:22:03.673198 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:03.673165    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="82a6a677-4ccf-4075-bf8b-5ce1d7f91586" containerName="agent"
Jul 28 10:22:03.673293 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:03.673218    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82a6a677-4ccf-4075-bf8b-5ce1d7f91586" containerName="kube-rbac-proxy"
Jul 28 10:22:03.673293 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:03.673224    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="82a6a677-4ccf-4075-bf8b-5ce1d7f91586" containerName="kube-rbac-proxy"
Jul 28 10:22:03.673293 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:03.673232    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82a6a677-4ccf-4075-bf8b-5ce1d7f91586" containerName="storage-initializer"
Jul 28 10:22:03.673293 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:03.673238    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="82a6a677-4ccf-4075-bf8b-5ce1d7f91586" containerName="storage-initializer"
Jul 28 10:22:03.673293 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:03.673286    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="82a6a677-4ccf-4075-bf8b-5ce1d7f91586" containerName="kserve-container"
Jul 28 10:22:03.673293 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:03.673294    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="82a6a677-4ccf-4075-bf8b-5ce1d7f91586" containerName="agent"
Jul 28 10:22:03.673474 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:03.673301    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="82a6a677-4ccf-4075-bf8b-5ce1d7f91586" containerName="kube-rbac-proxy"
Jul 28 10:22:03.676266 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:03.676246    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq"
Jul 28 10:22:03.679876 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:03.679710    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-b0501-kube-rbac-proxy-sar-config\""
Jul 28 10:22:03.679876 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:03.679716    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-b0501-predictor-serving-cert\""
Jul 28 10:22:03.691673 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:03.691654    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq"]
Jul 28 10:22:03.726202 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:03.726161    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:22:03.764525 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:03.764502    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt"]
Jul 28 10:22:03.767270 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:03.767254    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt"
Jul 28 10:22:03.769404 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:03.769377    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-b0501-predictor-serving-cert\""
Jul 28 10:22:03.769510 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:03.769491    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-b0501-kube-rbac-proxy-sar-config\""
Jul 28 10:22:03.769568 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:03.769509    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ff171d2-0800-4761-aaf9-193a083ed88b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq\" (UID: \"3ff171d2-0800-4761-aaf9-193a083ed88b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq"
Jul 28 10:22:03.769568 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:03.769551    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3ff171d2-0800-4761-aaf9-193a083ed88b-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq\" (UID: \"3ff171d2-0800-4761-aaf9-193a083ed88b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq"
Jul 28 10:22:03.769673 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:03.769583    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q8mg\" (UniqueName: \"kubernetes.io/projected/3ff171d2-0800-4761-aaf9-193a083ed88b-kube-api-access-5q8mg\") pod \"isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq\" (UID: \"3ff171d2-0800-4761-aaf9-193a083ed88b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq"
Jul 28 10:22:03.769673 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:03.769649    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3ff171d2-0800-4761-aaf9-193a083ed88b-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq\" (UID: \"3ff171d2-0800-4761-aaf9-193a083ed88b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq"
Jul 28 10:22:03.769775 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:03.769722    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ff171d2-0800-4761-aaf9-193a083ed88b-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq\" (UID: \"3ff171d2-0800-4761-aaf9-193a083ed88b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq"
Jul 28 10:22:03.779475 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:03.779456    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt"]
Jul 28 10:22:03.870505 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:03.870470    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q8mg\" (UniqueName: \"kubernetes.io/projected/3ff171d2-0800-4761-aaf9-193a083ed88b-kube-api-access-5q8mg\") pod \"isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq\" (UID: \"3ff171d2-0800-4761-aaf9-193a083ed88b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq"
Jul 28 10:22:03.870646 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:03.870544    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3ff171d2-0800-4761-aaf9-193a083ed88b-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq\" (UID: \"3ff171d2-0800-4761-aaf9-193a083ed88b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq"
Jul 28 10:22:03.870646 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:03.870579    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a48daa8-0991-4082-a471-629bb7afdf09-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt\" (UID: \"3a48daa8-0991-4082-a471-629bb7afdf09\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt"
Jul 28 10:22:03.870646 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:03.870633    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a48daa8-0991-4082-a471-629bb7afdf09-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt\" (UID: \"3a48daa8-0991-4082-a471-629bb7afdf09\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt"
Jul 28 10:22:03.870819 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:03.870662    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a48daa8-0991-4082-a471-629bb7afdf09-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt\" (UID: \"3a48daa8-0991-4082-a471-629bb7afdf09\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt"
Jul 28 10:22:03.870819 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:03.870706    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ff171d2-0800-4761-aaf9-193a083ed88b-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq\" (UID: \"3ff171d2-0800-4761-aaf9-193a083ed88b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq"
Jul 28 10:22:03.870819 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:03.870738    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a48daa8-0991-4082-a471-629bb7afdf09-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt\" (UID: \"3a48daa8-0991-4082-a471-629bb7afdf09\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt"
Jul 28 10:22:03.870819 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:03.870768    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drxx5\" (UniqueName: \"kubernetes.io/projected/3a48daa8-0991-4082-a471-629bb7afdf09-kube-api-access-drxx5\") pod \"isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt\" (UID: \"3a48daa8-0991-4082-a471-629bb7afdf09\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt"
Jul 28 10:22:03.870819 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:03.870802    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ff171d2-0800-4761-aaf9-193a083ed88b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq\" (UID: \"3ff171d2-0800-4761-aaf9-193a083ed88b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq"
Jul 28 10:22:03.871064 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:03.870834    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3ff171d2-0800-4761-aaf9-193a083ed88b-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq\" (UID: \"3ff171d2-0800-4761-aaf9-193a083ed88b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq"
Jul 28 10:22:03.871064 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:22:03.870839    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-b0501-predictor-serving-cert: secret "isvc-sklearn-graph-raw-hpa-b0501-predictor-serving-cert" not found
Jul 28 10:22:03.871064 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:22:03.870910    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ff171d2-0800-4761-aaf9-193a083ed88b-proxy-tls podName:3ff171d2-0800-4761-aaf9-193a083ed88b nodeName:}" failed. No retries permitted until 2026-07-28 10:22:04.370889152 +0000 UTC m=+863.471395697 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3ff171d2-0800-4761-aaf9-193a083ed88b-proxy-tls") pod "isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq" (UID: "3ff171d2-0800-4761-aaf9-193a083ed88b") : secret "isvc-sklearn-graph-raw-hpa-b0501-predictor-serving-cert" not found
Jul 28 10:22:03.871064 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:03.870960    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3ff171d2-0800-4761-aaf9-193a083ed88b-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq\" (UID: \"3ff171d2-0800-4761-aaf9-193a083ed88b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq"
Jul 28 10:22:03.871497 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:03.871480    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3ff171d2-0800-4761-aaf9-193a083ed88b-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq\" (UID: \"3ff171d2-0800-4761-aaf9-193a083ed88b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq"
Jul 28 10:22:03.871534 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:03.871505    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ff171d2-0800-4761-aaf9-193a083ed88b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq\" (UID: \"3ff171d2-0800-4761-aaf9-193a083ed88b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq"
Jul 28 10:22:03.877796 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:03.877771    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp"]
Jul 28 10:22:03.878072 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:03.878028    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp" podUID="285d0b61-573d-4069-82fd-f1a248dbb07f" containerName="kserve-container" containerID="cri-o://27c1fcb73b14ec1f454621cca50be756972a4b23ac3d56b5129bd2731782362e" gracePeriod=30
Jul 28 10:22:03.878144 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:03.878082    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp" podUID="285d0b61-573d-4069-82fd-f1a248dbb07f" containerName="kube-rbac-proxy" containerID="cri-o://2a2f93c9e79cc8bc530de384c50a2486f51d4cea2942a3b52b85e6989a7994c5" gracePeriod=30
Jul 28 10:22:03.882023 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:03.882002    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q8mg\" (UniqueName: \"kubernetes.io/projected/3ff171d2-0800-4761-aaf9-193a083ed88b-kube-api-access-5q8mg\") pod \"isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq\" (UID: \"3ff171d2-0800-4761-aaf9-193a083ed88b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq"
Jul 28 10:22:03.893323 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:03.893269    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:22:03.971412 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:03.971355    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a48daa8-0991-4082-a471-629bb7afdf09-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt\" (UID: \"3a48daa8-0991-4082-a471-629bb7afdf09\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt"
Jul 28 10:22:03.971412 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:03.971390    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a48daa8-0991-4082-a471-629bb7afdf09-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt\" (UID: \"3a48daa8-0991-4082-a471-629bb7afdf09\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt"
Jul 28 10:22:03.971412 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:03.971409    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a48daa8-0991-4082-a471-629bb7afdf09-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt\" (UID: \"3a48daa8-0991-4082-a471-629bb7afdf09\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt"
Jul 28 10:22:03.971615 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:03.971561    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a48daa8-0991-4082-a471-629bb7afdf09-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt\" (UID: \"3a48daa8-0991-4082-a471-629bb7afdf09\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt"
Jul 28 10:22:03.971615 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:03.971602    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drxx5\" (UniqueName: \"kubernetes.io/projected/3a48daa8-0991-4082-a471-629bb7afdf09-kube-api-access-drxx5\") pod \"isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt\" (UID: \"3a48daa8-0991-4082-a471-629bb7afdf09\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt"
Jul 28 10:22:03.971756 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:22:03.971738    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-b0501-predictor-serving-cert: secret "isvc-xgboost-graph-raw-hpa-b0501-predictor-serving-cert" not found
Jul 28 10:22:03.971813 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:03.971750    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a48daa8-0991-4082-a471-629bb7afdf09-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt\" (UID: \"3a48daa8-0991-4082-a471-629bb7afdf09\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt"
Jul 28 10:22:03.971813 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:22:03.971811    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a48daa8-0991-4082-a471-629bb7afdf09-proxy-tls podName:3a48daa8-0991-4082-a471-629bb7afdf09 nodeName:}" failed. No retries permitted until 2026-07-28 10:22:04.471790783 +0000 UTC m=+863.572297318 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3a48daa8-0991-4082-a471-629bb7afdf09-proxy-tls") pod "isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt" (UID: "3a48daa8-0991-4082-a471-629bb7afdf09") : secret "isvc-xgboost-graph-raw-hpa-b0501-predictor-serving-cert" not found
Jul 28 10:22:03.972088 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:03.972068    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a48daa8-0991-4082-a471-629bb7afdf09-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt\" (UID: \"3a48daa8-0991-4082-a471-629bb7afdf09\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt"
Jul 28 10:22:03.972132 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:03.972083    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a48daa8-0991-4082-a471-629bb7afdf09-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt\" (UID: \"3a48daa8-0991-4082-a471-629bb7afdf09\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt"
Jul 28 10:22:03.979404 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:03.979384    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drxx5\" (UniqueName: \"kubernetes.io/projected/3a48daa8-0991-4082-a471-629bb7afdf09-kube-api-access-drxx5\") pod \"isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt\" (UID: \"3a48daa8-0991-4082-a471-629bb7afdf09\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt"
Jul 28 10:22:03.992057 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:03.992036    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:22:04.138885 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:04.138857    2583 generic.go:356] "Generic (PLEG): container finished" podID="ecf844c8-0bcc-4e5a-8d75-301f9560c753" containerID="7642a0c012c7e5f04913b477af40ce75e168b8475a0b63977f372836f3f1d5f0" exitCode=2
Jul 28 10:22:04.139023 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:04.138936    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m" event={"ID":"ecf844c8-0bcc-4e5a-8d75-301f9560c753","Type":"ContainerDied","Data":"7642a0c012c7e5f04913b477af40ce75e168b8475a0b63977f372836f3f1d5f0"}
Jul 28 10:22:04.140983 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:04.140964    2583 generic.go:356] "Generic (PLEG): container finished" podID="285d0b61-573d-4069-82fd-f1a248dbb07f" containerID="2a2f93c9e79cc8bc530de384c50a2486f51d4cea2942a3b52b85e6989a7994c5" exitCode=2
Jul 28 10:22:04.141077 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:04.141034    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp" event={"ID":"285d0b61-573d-4069-82fd-f1a248dbb07f","Type":"ContainerDied","Data":"2a2f93c9e79cc8bc530de384c50a2486f51d4cea2942a3b52b85e6989a7994c5"}
Jul 28 10:22:04.141298 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:04.141280    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:22:04.374660 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:04.374594    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ff171d2-0800-4761-aaf9-193a083ed88b-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq\" (UID: \"3ff171d2-0800-4761-aaf9-193a083ed88b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq"
Jul 28 10:22:04.376832 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:04.376815    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ff171d2-0800-4761-aaf9-193a083ed88b-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq\" (UID: \"3ff171d2-0800-4761-aaf9-193a083ed88b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq"
Jul 28 10:22:04.474932 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:04.474909    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a48daa8-0991-4082-a471-629bb7afdf09-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt\" (UID: \"3a48daa8-0991-4082-a471-629bb7afdf09\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt"
Jul 28 10:22:04.476977 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:04.476960    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a48daa8-0991-4082-a471-629bb7afdf09-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt\" (UID: \"3a48daa8-0991-4082-a471-629bb7afdf09\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt"
Jul 28 10:22:04.589596 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:04.589557    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq"
Jul 28 10:22:04.676844 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:04.676814    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt"
Jul 28 10:22:04.724384 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:04.724361    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq"]
Jul 28 10:22:04.812232 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:04.812208    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt"]
Jul 28 10:22:04.813117 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:22:04.813097    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a48daa8_0991_4082_a471_629bb7afdf09.slice/crio-b790a5e8e42ec5b312ae7572064a958a5bfe305c51d2a4079e1f19d01e656d2c WatchSource:0}: Error finding container b790a5e8e42ec5b312ae7572064a958a5bfe305c51d2a4079e1f19d01e656d2c: Status 404 returned error can't find the container with id b790a5e8e42ec5b312ae7572064a958a5bfe305c51d2a4079e1f19d01e656d2c
Jul 28 10:22:05.147775 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:05.147743    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq" event={"ID":"3ff171d2-0800-4761-aaf9-193a083ed88b","Type":"ContainerStarted","Data":"5a7c0a84fe9fa3c520b471e932e0a44c89742597b9cb6e7feb4fff343478da74"}
Jul 28 10:22:05.148007 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:05.147805    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq" event={"ID":"3ff171d2-0800-4761-aaf9-193a083ed88b","Type":"ContainerStarted","Data":"3e7eaf23de02491455988e3b7781b56939380c5485f9ec82341d3c9195e3f916"}
Jul 28 10:22:05.149542 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:05.149518    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt" event={"ID":"3a48daa8-0991-4082-a471-629bb7afdf09","Type":"ContainerStarted","Data":"748ac6aaaf1b2a1fa052440ab56f31eaa251c49e04eda0df5a8338ed0b429298"}
Jul 28 10:22:05.149654 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:05.149561    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt" event={"ID":"3a48daa8-0991-4082-a471-629bb7afdf09","Type":"ContainerStarted","Data":"b790a5e8e42ec5b312ae7572064a958a5bfe305c51d2a4079e1f19d01e656d2c"}
Jul 28 10:22:05.649854 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:05.649817    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m" podUID="ecf844c8-0bcc-4e5a-8d75-301f9560c753" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Jul 28 10:22:05.654688 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:05.654656    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m" podUID="ecf844c8-0bcc-4e5a-8d75-301f9560c753" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 28 10:22:07.321753 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:07.321730    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp"
Jul 28 10:22:07.397876 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:07.397850    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/285d0b61-573d-4069-82fd-f1a248dbb07f-cabundle-cert\") pod \"285d0b61-573d-4069-82fd-f1a248dbb07f\" (UID: \"285d0b61-573d-4069-82fd-f1a248dbb07f\") "
Jul 28 10:22:07.398002 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:07.397902    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/285d0b61-573d-4069-82fd-f1a248dbb07f-kube-rbac-proxy-sar-config\") pod \"285d0b61-573d-4069-82fd-f1a248dbb07f\" (UID: \"285d0b61-573d-4069-82fd-f1a248dbb07f\") "
Jul 28 10:22:07.398002 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:07.397933    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nk5th\" (UniqueName: \"kubernetes.io/projected/285d0b61-573d-4069-82fd-f1a248dbb07f-kube-api-access-nk5th\") pod \"285d0b61-573d-4069-82fd-f1a248dbb07f\" (UID: \"285d0b61-573d-4069-82fd-f1a248dbb07f\") "
Jul 28 10:22:07.398002 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:07.397964    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/285d0b61-573d-4069-82fd-f1a248dbb07f-proxy-tls\") pod \"285d0b61-573d-4069-82fd-f1a248dbb07f\" (UID: \"285d0b61-573d-4069-82fd-f1a248dbb07f\") "
Jul 28 10:22:07.398002 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:07.398000    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/285d0b61-573d-4069-82fd-f1a248dbb07f-kserve-provision-location\") pod \"285d0b61-573d-4069-82fd-f1a248dbb07f\" (UID: \"285d0b61-573d-4069-82fd-f1a248dbb07f\") "
Jul 28 10:22:07.398285 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:07.398249    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/285d0b61-573d-4069-82fd-f1a248dbb07f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "285d0b61-573d-4069-82fd-f1a248dbb07f" (UID: "285d0b61-573d-4069-82fd-f1a248dbb07f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:22:07.398379 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:07.398357    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/285d0b61-573d-4069-82fd-f1a248dbb07f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "285d0b61-573d-4069-82fd-f1a248dbb07f" (UID: "285d0b61-573d-4069-82fd-f1a248dbb07f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:22:07.398420 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:07.398381    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/285d0b61-573d-4069-82fd-f1a248dbb07f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "285d0b61-573d-4069-82fd-f1a248dbb07f" (UID: "285d0b61-573d-4069-82fd-f1a248dbb07f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:22:07.399965 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:07.399936    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/285d0b61-573d-4069-82fd-f1a248dbb07f-kube-api-access-nk5th" (OuterVolumeSpecName: "kube-api-access-nk5th") pod "285d0b61-573d-4069-82fd-f1a248dbb07f" (UID: "285d0b61-573d-4069-82fd-f1a248dbb07f"). InnerVolumeSpecName "kube-api-access-nk5th". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:22:07.400056 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:07.400016    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/285d0b61-573d-4069-82fd-f1a248dbb07f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "285d0b61-573d-4069-82fd-f1a248dbb07f" (UID: "285d0b61-573d-4069-82fd-f1a248dbb07f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:22:07.499427 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:07.499385    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/285d0b61-573d-4069-82fd-f1a248dbb07f-cabundle-cert\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:22:07.499427 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:07.499423    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/285d0b61-573d-4069-82fd-f1a248dbb07f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:22:07.499587 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:07.499438    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nk5th\" (UniqueName: \"kubernetes.io/projected/285d0b61-573d-4069-82fd-f1a248dbb07f-kube-api-access-nk5th\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:22:07.499587 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:07.499451    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/285d0b61-573d-4069-82fd-f1a248dbb07f-proxy-tls\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:22:07.499587 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:07.499463    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/285d0b61-573d-4069-82fd-f1a248dbb07f-kserve-provision-location\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:22:08.047351 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:08.047329    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m"
Jul 28 10:22:08.105224 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:08.105139    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ecf844c8-0bcc-4e5a-8d75-301f9560c753-proxy-tls\") pod \"ecf844c8-0bcc-4e5a-8d75-301f9560c753\" (UID: \"ecf844c8-0bcc-4e5a-8d75-301f9560c753\") "
Jul 28 10:22:08.105352 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:08.105247    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ecf844c8-0bcc-4e5a-8d75-301f9560c753-kserve-provision-location\") pod \"ecf844c8-0bcc-4e5a-8d75-301f9560c753\" (UID: \"ecf844c8-0bcc-4e5a-8d75-301f9560c753\") "
Jul 28 10:22:08.105352 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:08.105290    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ecf844c8-0bcc-4e5a-8d75-301f9560c753-kube-rbac-proxy-sar-config\") pod \"ecf844c8-0bcc-4e5a-8d75-301f9560c753\" (UID: \"ecf844c8-0bcc-4e5a-8d75-301f9560c753\") "
Jul 28 10:22:08.105352 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:08.105336    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zw8t2\" (UniqueName: \"kubernetes.io/projected/ecf844c8-0bcc-4e5a-8d75-301f9560c753-kube-api-access-zw8t2\") pod \"ecf844c8-0bcc-4e5a-8d75-301f9560c753\" (UID: \"ecf844c8-0bcc-4e5a-8d75-301f9560c753\") "
Jul 28 10:22:08.105515 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:08.105367    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ecf844c8-0bcc-4e5a-8d75-301f9560c753-cabundle-cert\") pod \"ecf844c8-0bcc-4e5a-8d75-301f9560c753\" (UID: \"ecf844c8-0bcc-4e5a-8d75-301f9560c753\") "
Jul 28 10:22:08.106152 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:08.105831    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecf844c8-0bcc-4e5a-8d75-301f9560c753-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ecf844c8-0bcc-4e5a-8d75-301f9560c753" (UID: "ecf844c8-0bcc-4e5a-8d75-301f9560c753"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:22:08.106303 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:08.106205    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecf844c8-0bcc-4e5a-8d75-301f9560c753-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ecf844c8-0bcc-4e5a-8d75-301f9560c753" (UID: "ecf844c8-0bcc-4e5a-8d75-301f9560c753"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:22:08.106303 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:08.106276    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecf844c8-0bcc-4e5a-8d75-301f9560c753-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ecf844c8-0bcc-4e5a-8d75-301f9560c753" (UID: "ecf844c8-0bcc-4e5a-8d75-301f9560c753"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:22:08.108548 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:08.108519    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecf844c8-0bcc-4e5a-8d75-301f9560c753-kube-api-access-zw8t2" (OuterVolumeSpecName: "kube-api-access-zw8t2") pod "ecf844c8-0bcc-4e5a-8d75-301f9560c753" (UID: "ecf844c8-0bcc-4e5a-8d75-301f9560c753"). InnerVolumeSpecName "kube-api-access-zw8t2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:22:08.108677 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:08.108620    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecf844c8-0bcc-4e5a-8d75-301f9560c753-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ecf844c8-0bcc-4e5a-8d75-301f9560c753" (UID: "ecf844c8-0bcc-4e5a-8d75-301f9560c753"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:22:08.164972 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:08.164941    2583 generic.go:356] "Generic (PLEG): container finished" podID="285d0b61-573d-4069-82fd-f1a248dbb07f" containerID="27c1fcb73b14ec1f454621cca50be756972a4b23ac3d56b5129bd2731782362e" exitCode=0
Jul 28 10:22:08.165088 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:08.164978    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp" event={"ID":"285d0b61-573d-4069-82fd-f1a248dbb07f","Type":"ContainerDied","Data":"27c1fcb73b14ec1f454621cca50be756972a4b23ac3d56b5129bd2731782362e"}
Jul 28 10:22:08.165088 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:08.165005    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp"
Jul 28 10:22:08.165088 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:08.165083    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp" event={"ID":"285d0b61-573d-4069-82fd-f1a248dbb07f","Type":"ContainerDied","Data":"0de34e18dc5f5cd744d16be7a58a38275db5a8c07591ff893decfc40ef5055b3"}
Jul 28 10:22:08.165273 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:08.165128    2583 scope.go:117] "RemoveContainer" containerID="2a2f93c9e79cc8bc530de384c50a2486f51d4cea2942a3b52b85e6989a7994c5"
Jul 28 10:22:08.167283 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:08.167253    2583 generic.go:356] "Generic (PLEG): container finished" podID="ecf844c8-0bcc-4e5a-8d75-301f9560c753" containerID="5ff08d35b66c6d1b9b15e748e1d7c050d608fcb23aa37d753951fdf70794e6fa" exitCode=0
Jul 28 10:22:08.167385 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:08.167310    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m" event={"ID":"ecf844c8-0bcc-4e5a-8d75-301f9560c753","Type":"ContainerDied","Data":"5ff08d35b66c6d1b9b15e748e1d7c050d608fcb23aa37d753951fdf70794e6fa"}
Jul 28 10:22:08.167385 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:08.167371    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m"
Jul 28 10:22:08.167497 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:08.167395    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m" event={"ID":"ecf844c8-0bcc-4e5a-8d75-301f9560c753","Type":"ContainerDied","Data":"60ad422c7427c10924260fd5aa9629bd1790ed1d01e01910f4959c9734c839a2"}
Jul 28 10:22:08.174812 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:08.174793    2583 scope.go:117] "RemoveContainer" containerID="27c1fcb73b14ec1f454621cca50be756972a4b23ac3d56b5129bd2731782362e"
Jul 28 10:22:08.182693 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:08.182672    2583 scope.go:117] "RemoveContainer" containerID="db07a8689b8da2639161cf393883ae59fa21c14f9609d44c1339bd9e30862a30"
Jul 28 10:22:08.183570 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:08.183553    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp"]
Jul 28 10:22:08.189423 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:08.189403    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-74e2d-predictor-78b98d877c-jmzmp"]
Jul 28 10:22:08.190217 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:08.190200    2583 scope.go:117] "RemoveContainer" containerID="2a2f93c9e79cc8bc530de384c50a2486f51d4cea2942a3b52b85e6989a7994c5"
Jul 28 10:22:08.190457 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:22:08.190439    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a2f93c9e79cc8bc530de384c50a2486f51d4cea2942a3b52b85e6989a7994c5\": container with ID starting with 2a2f93c9e79cc8bc530de384c50a2486f51d4cea2942a3b52b85e6989a7994c5 not found: ID does not exist" containerID="2a2f93c9e79cc8bc530de384c50a2486f51d4cea2942a3b52b85e6989a7994c5"
Jul 28 10:22:08.190510 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:08.190466    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a2f93c9e79cc8bc530de384c50a2486f51d4cea2942a3b52b85e6989a7994c5"} err="failed to get container status \"2a2f93c9e79cc8bc530de384c50a2486f51d4cea2942a3b52b85e6989a7994c5\": rpc error: code = NotFound desc = could not find container \"2a2f93c9e79cc8bc530de384c50a2486f51d4cea2942a3b52b85e6989a7994c5\": container with ID starting with 2a2f93c9e79cc8bc530de384c50a2486f51d4cea2942a3b52b85e6989a7994c5 not found: ID does not exist"
Jul 28 10:22:08.190510 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:08.190484    2583 scope.go:117] "RemoveContainer" containerID="27c1fcb73b14ec1f454621cca50be756972a4b23ac3d56b5129bd2731782362e"
Jul 28 10:22:08.190672 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:22:08.190660    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27c1fcb73b14ec1f454621cca50be756972a4b23ac3d56b5129bd2731782362e\": container with ID starting with 27c1fcb73b14ec1f454621cca50be756972a4b23ac3d56b5129bd2731782362e not found: ID does not exist" containerID="27c1fcb73b14ec1f454621cca50be756972a4b23ac3d56b5129bd2731782362e"
Jul 28 10:22:08.190713 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:08.190675    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27c1fcb73b14ec1f454621cca50be756972a4b23ac3d56b5129bd2731782362e"} err="failed to get container status \"27c1fcb73b14ec1f454621cca50be756972a4b23ac3d56b5129bd2731782362e\": rpc error: code = NotFound desc = could not find container \"27c1fcb73b14ec1f454621cca50be756972a4b23ac3d56b5129bd2731782362e\": container with ID starting with 27c1fcb73b14ec1f454621cca50be756972a4b23ac3d56b5129bd2731782362e not found: ID does not exist"
Jul 28 10:22:08.190713 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:08.190687    2583 scope.go:117] "RemoveContainer" containerID="db07a8689b8da2639161cf393883ae59fa21c14f9609d44c1339bd9e30862a30"
Jul 28 10:22:08.190881 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:22:08.190863    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db07a8689b8da2639161cf393883ae59fa21c14f9609d44c1339bd9e30862a30\": container with ID starting with db07a8689b8da2639161cf393883ae59fa21c14f9609d44c1339bd9e30862a30 not found: ID does not exist" containerID="db07a8689b8da2639161cf393883ae59fa21c14f9609d44c1339bd9e30862a30"
Jul 28 10:22:08.190920 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:08.190886    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db07a8689b8da2639161cf393883ae59fa21c14f9609d44c1339bd9e30862a30"} err="failed to get container status \"db07a8689b8da2639161cf393883ae59fa21c14f9609d44c1339bd9e30862a30\": rpc error: code = NotFound desc = could not find container \"db07a8689b8da2639161cf393883ae59fa21c14f9609d44c1339bd9e30862a30\": container with ID starting with db07a8689b8da2639161cf393883ae59fa21c14f9609d44c1339bd9e30862a30 not found: ID does not exist"
Jul 28 10:22:08.190920 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:08.190901    2583 scope.go:117] "RemoveContainer" containerID="7642a0c012c7e5f04913b477af40ce75e168b8475a0b63977f372836f3f1d5f0"
Jul 28 10:22:08.197920 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:08.197784    2583 scope.go:117] "RemoveContainer" containerID="5ff08d35b66c6d1b9b15e748e1d7c050d608fcb23aa37d753951fdf70794e6fa"
Jul 28 10:22:08.203465 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:08.203445    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m"]
Jul 28 10:22:08.205222 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:08.205119    2583 scope.go:117] "RemoveContainer" containerID="64d7e07f34a87c2fc8a0626318fc7872c4335866e445c07f3c6d5c075f427664"
Jul 28 10:22:08.207357 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:08.207330    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-74e2d-predictor-577d9b7b69-fdl4m"]
Jul 28 10:22:08.207655 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:08.207637    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zw8t2\" (UniqueName: \"kubernetes.io/projected/ecf844c8-0bcc-4e5a-8d75-301f9560c753-kube-api-access-zw8t2\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:22:08.207694 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:08.207661    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ecf844c8-0bcc-4e5a-8d75-301f9560c753-cabundle-cert\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:22:08.207694 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:08.207671    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ecf844c8-0bcc-4e5a-8d75-301f9560c753-proxy-tls\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:22:08.207694 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:08.207679    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ecf844c8-0bcc-4e5a-8d75-301f9560c753-kserve-provision-location\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:22:08.207694 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:08.207687    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ecf844c8-0bcc-4e5a-8d75-301f9560c753-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:22:08.212362 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:08.212348    2583 scope.go:117] "RemoveContainer" containerID="7642a0c012c7e5f04913b477af40ce75e168b8475a0b63977f372836f3f1d5f0"
Jul 28 10:22:08.212584 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:22:08.212567    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7642a0c012c7e5f04913b477af40ce75e168b8475a0b63977f372836f3f1d5f0\": container with ID starting with 7642a0c012c7e5f04913b477af40ce75e168b8475a0b63977f372836f3f1d5f0 not found: ID does not exist" containerID="7642a0c012c7e5f04913b477af40ce75e168b8475a0b63977f372836f3f1d5f0"
Jul 28 10:22:08.212626 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:08.212590    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7642a0c012c7e5f04913b477af40ce75e168b8475a0b63977f372836f3f1d5f0"} err="failed to get container status \"7642a0c012c7e5f04913b477af40ce75e168b8475a0b63977f372836f3f1d5f0\": rpc error: code = NotFound desc = could not find container \"7642a0c012c7e5f04913b477af40ce75e168b8475a0b63977f372836f3f1d5f0\": container with ID starting with 7642a0c012c7e5f04913b477af40ce75e168b8475a0b63977f372836f3f1d5f0 not found: ID does not exist"
Jul 28 10:22:08.212626 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:08.212607    2583 scope.go:117] "RemoveContainer" containerID="5ff08d35b66c6d1b9b15e748e1d7c050d608fcb23aa37d753951fdf70794e6fa"
Jul 28 10:22:08.212815 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:22:08.212800    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ff08d35b66c6d1b9b15e748e1d7c050d608fcb23aa37d753951fdf70794e6fa\": container with ID starting with 5ff08d35b66c6d1b9b15e748e1d7c050d608fcb23aa37d753951fdf70794e6fa not found: ID does not exist" containerID="5ff08d35b66c6d1b9b15e748e1d7c050d608fcb23aa37d753951fdf70794e6fa"
Jul 28 10:22:08.212858 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:08.212819    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ff08d35b66c6d1b9b15e748e1d7c050d608fcb23aa37d753951fdf70794e6fa"} err="failed to get container status \"5ff08d35b66c6d1b9b15e748e1d7c050d608fcb23aa37d753951fdf70794e6fa\": rpc error: code = NotFound desc = could not find container \"5ff08d35b66c6d1b9b15e748e1d7c050d608fcb23aa37d753951fdf70794e6fa\": container with ID starting with 5ff08d35b66c6d1b9b15e748e1d7c050d608fcb23aa37d753951fdf70794e6fa not found: ID does not exist"
Jul 28 10:22:08.212858 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:08.212831    2583 scope.go:117] "RemoveContainer" containerID="64d7e07f34a87c2fc8a0626318fc7872c4335866e445c07f3c6d5c075f427664"
Jul 28 10:22:08.213033 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:22:08.213018    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64d7e07f34a87c2fc8a0626318fc7872c4335866e445c07f3c6d5c075f427664\": container with ID starting with 64d7e07f34a87c2fc8a0626318fc7872c4335866e445c07f3c6d5c075f427664 not found: ID does not exist" containerID="64d7e07f34a87c2fc8a0626318fc7872c4335866e445c07f3c6d5c075f427664"
Jul 28 10:22:08.213072 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:08.213037    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64d7e07f34a87c2fc8a0626318fc7872c4335866e445c07f3c6d5c075f427664"} err="failed to get container status \"64d7e07f34a87c2fc8a0626318fc7872c4335866e445c07f3c6d5c075f427664\": rpc error: code = NotFound desc = could not find container \"64d7e07f34a87c2fc8a0626318fc7872c4335866e445c07f3c6d5c075f427664\": container with ID starting with 64d7e07f34a87c2fc8a0626318fc7872c4335866e445c07f3c6d5c075f427664 not found: ID does not exist"
Jul 28 10:22:09.173759 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:09.173728    2583 generic.go:356] "Generic (PLEG): container finished" podID="3a48daa8-0991-4082-a471-629bb7afdf09" containerID="748ac6aaaf1b2a1fa052440ab56f31eaa251c49e04eda0df5a8338ed0b429298" exitCode=0
Jul 28 10:22:09.174234 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:09.173806    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt" event={"ID":"3a48daa8-0991-4082-a471-629bb7afdf09","Type":"ContainerDied","Data":"748ac6aaaf1b2a1fa052440ab56f31eaa251c49e04eda0df5a8338ed0b429298"}
Jul 28 10:22:09.175352 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:09.175321    2583 generic.go:356] "Generic (PLEG): container finished" podID="3ff171d2-0800-4761-aaf9-193a083ed88b" containerID="5a7c0a84fe9fa3c520b471e932e0a44c89742597b9cb6e7feb4fff343478da74" exitCode=0
Jul 28 10:22:09.175436 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:09.175376    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq" event={"ID":"3ff171d2-0800-4761-aaf9-193a083ed88b","Type":"ContainerDied","Data":"5a7c0a84fe9fa3c520b471e932e0a44c89742597b9cb6e7feb4fff343478da74"}
Jul 28 10:22:09.440762 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:09.440687    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="285d0b61-573d-4069-82fd-f1a248dbb07f" path="/var/lib/kubelet/pods/285d0b61-573d-4069-82fd-f1a248dbb07f/volumes"
Jul 28 10:22:09.441255 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:09.441239    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ecf844c8-0bcc-4e5a-8d75-301f9560c753" path="/var/lib/kubelet/pods/ecf844c8-0bcc-4e5a-8d75-301f9560c753/volumes"
Jul 28 10:22:10.184054 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:10.184019    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt" event={"ID":"3a48daa8-0991-4082-a471-629bb7afdf09","Type":"ContainerStarted","Data":"4ab09614b0452242a1f574cefc9eca19a13e75e7ec98028e79346655d14ca0d0"}
Jul 28 10:22:10.184474 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:10.184082    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt" event={"ID":"3a48daa8-0991-4082-a471-629bb7afdf09","Type":"ContainerStarted","Data":"cbb064ef3c18f48638fdee6960ee7e725147b9d71358ec39360efce929cc2b5e"}
Jul 28 10:22:10.184474 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:10.184285    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt"
Jul 28 10:22:10.186098 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:10.186077    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq" event={"ID":"3ff171d2-0800-4761-aaf9-193a083ed88b","Type":"ContainerStarted","Data":"37a7f64c03a3bd2bb2e08debff56aaa172ae8ac5c1b36f77cf9bbdc0a15e8576"}
Jul 28 10:22:10.186225 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:10.186120    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq" event={"ID":"3ff171d2-0800-4761-aaf9-193a083ed88b","Type":"ContainerStarted","Data":"7a4cab835439f808773d7e28725d98dd03df9236363739eae82d1fe40427b958"}
Jul 28 10:22:10.186386 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:10.186369    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq"
Jul 28 10:22:10.186455 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:10.186424    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq"
Jul 28 10:22:10.187624 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:10.187599    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq" podUID="3ff171d2-0800-4761-aaf9-193a083ed88b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 10:22:10.203140 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:10.203101    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt" podStartSLOduration=3.425838549 podStartE2EDuration="7.203088152s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.777249603s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:22:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:22:10.201584799 +0000 UTC m=+869.302091352" watchObservedRunningTime="2026-07-28 10:22:10.203088152 +0000 UTC m=+869.303594705"
Jul 28 10:22:10.221095 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:10.221046    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq" podStartSLOduration=3.261177586 podStartE2EDuration="7.221034166s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.95985658s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:22:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:22:10.218937449 +0000 UTC m=+869.319444015" watchObservedRunningTime="2026-07-28 10:22:10.221034166 +0000 UTC m=+869.321540720"
Jul 28 10:22:11.191667 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:11.191629    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq" podUID="3ff171d2-0800-4761-aaf9-193a083ed88b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 10:22:11.192089 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:11.191822    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt"
Jul 28 10:22:11.193077 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:11.193049    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt" podUID="3a48daa8-0991-4082-a471-629bb7afdf09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 10:22:12.196076 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:12.196031    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt" podUID="3a48daa8-0991-4082-a471-629bb7afdf09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 10:22:16.196486 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:16.196460    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq"
Jul 28 10:22:16.196974 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:16.196947    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq" podUID="3ff171d2-0800-4761-aaf9-193a083ed88b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 10:22:17.200663 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:17.200631    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt"
Jul 28 10:22:17.201171 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:17.201142    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt" podUID="3a48daa8-0991-4082-a471-629bb7afdf09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 10:22:26.197076 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:26.197039    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq" podUID="3ff171d2-0800-4761-aaf9-193a083ed88b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 10:22:27.201798 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:27.201751    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt" podUID="3a48daa8-0991-4082-a471-629bb7afdf09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 10:22:36.197994 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:36.197902    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq" podUID="3ff171d2-0800-4761-aaf9-193a083ed88b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 10:22:37.201561 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:37.201523    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt" podUID="3a48daa8-0991-4082-a471-629bb7afdf09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 10:22:46.197673 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:46.197632    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq" podUID="3ff171d2-0800-4761-aaf9-193a083ed88b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 10:22:47.201445 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:47.201405    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt" podUID="3a48daa8-0991-4082-a471-629bb7afdf09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 10:22:52.183467 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:52.183435    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:22:56.197895 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:56.197861    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq" podUID="3ff171d2-0800-4761-aaf9-193a083ed88b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 10:22:57.202016 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:22:57.201973    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt" podUID="3a48daa8-0991-4082-a471-629bb7afdf09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 10:23:06.197644 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:06.197608    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq" podUID="3ff171d2-0800-4761-aaf9-193a083ed88b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 10:23:07.201338 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:07.201306    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt"
Jul 28 10:23:16.197755 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:16.197715    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq" podUID="3ff171d2-0800-4761-aaf9-193a083ed88b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 10:23:24.434207 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:24.434142    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq"
Jul 28 10:23:43.675111 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:43.675079    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:23:43.984478 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:43.984401    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:23:53.783643 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:53.783607    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:23:53.943164 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:53.943131    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:23:53.989073 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:53.989042    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-8e237-predictor-8c874fbcc-fnrsp"]
Jul 28 10:23:53.989625 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:53.989604    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="285d0b61-573d-4069-82fd-f1a248dbb07f" containerName="storage-initializer"
Jul 28 10:23:53.989789 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:53.989769    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="285d0b61-573d-4069-82fd-f1a248dbb07f" containerName="storage-initializer"
Jul 28 10:23:53.989789 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:53.989790    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecf844c8-0bcc-4e5a-8d75-301f9560c753" containerName="kube-rbac-proxy"
Jul 28 10:23:53.989789 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:53.989799    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecf844c8-0bcc-4e5a-8d75-301f9560c753" containerName="kube-rbac-proxy"
Jul 28 10:23:53.989789 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:53.989813    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecf844c8-0bcc-4e5a-8d75-301f9560c753" containerName="kserve-container"
Jul 28 10:23:53.990110 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:53.989822    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecf844c8-0bcc-4e5a-8d75-301f9560c753" containerName="kserve-container"
Jul 28 10:23:53.990110 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:53.989842    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="285d0b61-573d-4069-82fd-f1a248dbb07f" containerName="kserve-container"
Jul 28 10:23:53.990110 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:53.989850    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="285d0b61-573d-4069-82fd-f1a248dbb07f" containerName="kserve-container"
Jul 28 10:23:53.990110 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:53.989864    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="285d0b61-573d-4069-82fd-f1a248dbb07f" containerName="kube-rbac-proxy"
Jul 28 10:23:53.990110 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:53.989872    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="285d0b61-573d-4069-82fd-f1a248dbb07f" containerName="kube-rbac-proxy"
Jul 28 10:23:53.990110 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:53.989890    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecf844c8-0bcc-4e5a-8d75-301f9560c753" containerName="storage-initializer"
Jul 28 10:23:53.990110 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:53.989898    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecf844c8-0bcc-4e5a-8d75-301f9560c753" containerName="storage-initializer"
Jul 28 10:23:53.990110 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:53.989978    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="285d0b61-573d-4069-82fd-f1a248dbb07f" containerName="kube-rbac-proxy"
Jul 28 10:23:53.990110 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:53.989989    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="285d0b61-573d-4069-82fd-f1a248dbb07f" containerName="kserve-container"
Jul 28 10:23:53.990110 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:53.990001    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecf844c8-0bcc-4e5a-8d75-301f9560c753" containerName="kserve-container"
Jul 28 10:23:53.990110 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:53.990011    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecf844c8-0bcc-4e5a-8d75-301f9560c753" containerName="kube-rbac-proxy"
Jul 28 10:23:53.993434 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:53.993414    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-8e237-predictor-8c874fbcc-fnrsp"
Jul 28 10:23:53.995846 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:53.995823    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-8e237-kube-rbac-proxy-sar-config\""
Jul 28 10:23:53.995980 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:53.995937    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-8e237-predictor-serving-cert\""
Jul 28 10:23:54.002884 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:54.002519    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-8e237-predictor-8c874fbcc-fnrsp"]
Jul 28 10:23:54.019859 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:54.019822    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:23:54.025115 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:54.025092    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq"]
Jul 28 10:23:54.025361 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:54.025338    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq" podUID="3ff171d2-0800-4761-aaf9-193a083ed88b" containerName="kserve-container" containerID="cri-o://7a4cab835439f808773d7e28725d98dd03df9236363739eae82d1fe40427b958" gracePeriod=30
Jul 28 10:23:54.025454 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:54.025390    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq" podUID="3ff171d2-0800-4761-aaf9-193a083ed88b" containerName="kube-rbac-proxy" containerID="cri-o://37a7f64c03a3bd2bb2e08debff56aaa172ae8ac5c1b36f77cf9bbdc0a15e8576" gracePeriod=30
Jul 28 10:23:54.032665 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:54.032644    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt"]
Jul 28 10:23:54.032851 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:54.032832    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt" podUID="3a48daa8-0991-4082-a471-629bb7afdf09" containerName="kserve-container" containerID="cri-o://cbb064ef3c18f48638fdee6960ee7e725147b9d71358ec39360efce929cc2b5e" gracePeriod=30
Jul 28 10:23:54.032917 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:54.032901    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt" podUID="3a48daa8-0991-4082-a471-629bb7afdf09" containerName="kube-rbac-proxy" containerID="cri-o://4ab09614b0452242a1f574cefc9eca19a13e75e7ec98028e79346655d14ca0d0" gracePeriod=30
Jul 28 10:23:54.055428 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:54.055407    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86f2d777-6f0c-445d-b9f1-d6b43e8ef110-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-8e237-predictor-8c874fbcc-fnrsp\" (UID: \"86f2d777-6f0c-445d-b9f1-d6b43e8ef110\") " pod="kserve-ci-e2e-test/message-dumper-raw-8e237-predictor-8c874fbcc-fnrsp"
Jul 28 10:23:54.055534 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:54.055473    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86f2d777-6f0c-445d-b9f1-d6b43e8ef110-proxy-tls\") pod \"message-dumper-raw-8e237-predictor-8c874fbcc-fnrsp\" (UID: \"86f2d777-6f0c-445d-b9f1-d6b43e8ef110\") " pod="kserve-ci-e2e-test/message-dumper-raw-8e237-predictor-8c874fbcc-fnrsp"
Jul 28 10:23:54.055604 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:54.055532    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ccnz\" (UniqueName: \"kubernetes.io/projected/86f2d777-6f0c-445d-b9f1-d6b43e8ef110-kube-api-access-2ccnz\") pod \"message-dumper-raw-8e237-predictor-8c874fbcc-fnrsp\" (UID: \"86f2d777-6f0c-445d-b9f1-d6b43e8ef110\") " pod="kserve-ci-e2e-test/message-dumper-raw-8e237-predictor-8c874fbcc-fnrsp"
Jul 28 10:23:54.069816 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:54.069787    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:23:54.155895 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:54.155873    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ccnz\" (UniqueName: \"kubernetes.io/projected/86f2d777-6f0c-445d-b9f1-d6b43e8ef110-kube-api-access-2ccnz\") pod \"message-dumper-raw-8e237-predictor-8c874fbcc-fnrsp\" (UID: \"86f2d777-6f0c-445d-b9f1-d6b43e8ef110\") " pod="kserve-ci-e2e-test/message-dumper-raw-8e237-predictor-8c874fbcc-fnrsp"
Jul 28 10:23:54.155975 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:54.155928    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86f2d777-6f0c-445d-b9f1-d6b43e8ef110-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-8e237-predictor-8c874fbcc-fnrsp\" (UID: \"86f2d777-6f0c-445d-b9f1-d6b43e8ef110\") " pod="kserve-ci-e2e-test/message-dumper-raw-8e237-predictor-8c874fbcc-fnrsp"
Jul 28 10:23:54.155975 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:54.155958    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86f2d777-6f0c-445d-b9f1-d6b43e8ef110-proxy-tls\") pod \"message-dumper-raw-8e237-predictor-8c874fbcc-fnrsp\" (UID: \"86f2d777-6f0c-445d-b9f1-d6b43e8ef110\") " pod="kserve-ci-e2e-test/message-dumper-raw-8e237-predictor-8c874fbcc-fnrsp"
Jul 28 10:23:54.156080 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:23:54.156036    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-raw-8e237-predictor-serving-cert: secret "message-dumper-raw-8e237-predictor-serving-cert" not found
Jul 28 10:23:54.156132 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:23:54.156090    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86f2d777-6f0c-445d-b9f1-d6b43e8ef110-proxy-tls podName:86f2d777-6f0c-445d-b9f1-d6b43e8ef110 nodeName:}" failed. No retries permitted until 2026-07-28 10:23:54.656072543 +0000 UTC m=+973.756579075 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/86f2d777-6f0c-445d-b9f1-d6b43e8ef110-proxy-tls") pod "message-dumper-raw-8e237-predictor-8c874fbcc-fnrsp" (UID: "86f2d777-6f0c-445d-b9f1-d6b43e8ef110") : secret "message-dumper-raw-8e237-predictor-serving-cert" not found
Jul 28 10:23:54.156524 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:54.156505    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86f2d777-6f0c-445d-b9f1-d6b43e8ef110-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-8e237-predictor-8c874fbcc-fnrsp\" (UID: \"86f2d777-6f0c-445d-b9f1-d6b43e8ef110\") " pod="kserve-ci-e2e-test/message-dumper-raw-8e237-predictor-8c874fbcc-fnrsp"
Jul 28 10:23:54.166075 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:54.166053    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ccnz\" (UniqueName: \"kubernetes.io/projected/86f2d777-6f0c-445d-b9f1-d6b43e8ef110-kube-api-access-2ccnz\") pod \"message-dumper-raw-8e237-predictor-8c874fbcc-fnrsp\" (UID: \"86f2d777-6f0c-445d-b9f1-d6b43e8ef110\") " pod="kserve-ci-e2e-test/message-dumper-raw-8e237-predictor-8c874fbcc-fnrsp"
Jul 28 10:23:54.355239 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:54.355140    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:23:54.434142 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:54.434108    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq" podUID="3ff171d2-0800-4761-aaf9-193a083ed88b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 10:23:54.648355 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:54.648322    2583 generic.go:356] "Generic (PLEG): container finished" podID="3ff171d2-0800-4761-aaf9-193a083ed88b" containerID="37a7f64c03a3bd2bb2e08debff56aaa172ae8ac5c1b36f77cf9bbdc0a15e8576" exitCode=2
Jul 28 10:23:54.648494 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:54.648390    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq" event={"ID":"3ff171d2-0800-4761-aaf9-193a083ed88b","Type":"ContainerDied","Data":"37a7f64c03a3bd2bb2e08debff56aaa172ae8ac5c1b36f77cf9bbdc0a15e8576"}
Jul 28 10:23:54.650428 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:54.650409    2583 generic.go:356] "Generic (PLEG): container finished" podID="3a48daa8-0991-4082-a471-629bb7afdf09" containerID="4ab09614b0452242a1f574cefc9eca19a13e75e7ec98028e79346655d14ca0d0" exitCode=2
Jul 28 10:23:54.650530 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:54.650436    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt" event={"ID":"3a48daa8-0991-4082-a471-629bb7afdf09","Type":"ContainerDied","Data":"4ab09614b0452242a1f574cefc9eca19a13e75e7ec98028e79346655d14ca0d0"}
Jul 28 10:23:54.659725 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:54.659703    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86f2d777-6f0c-445d-b9f1-d6b43e8ef110-proxy-tls\") pod \"message-dumper-raw-8e237-predictor-8c874fbcc-fnrsp\" (UID: \"86f2d777-6f0c-445d-b9f1-d6b43e8ef110\") " pod="kserve-ci-e2e-test/message-dumper-raw-8e237-predictor-8c874fbcc-fnrsp"
Jul 28 10:23:54.662058 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:54.662034    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86f2d777-6f0c-445d-b9f1-d6b43e8ef110-proxy-tls\") pod \"message-dumper-raw-8e237-predictor-8c874fbcc-fnrsp\" (UID: \"86f2d777-6f0c-445d-b9f1-d6b43e8ef110\") " pod="kserve-ci-e2e-test/message-dumper-raw-8e237-predictor-8c874fbcc-fnrsp"
Jul 28 10:23:54.907716 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:54.907646    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-8e237-predictor-8c874fbcc-fnrsp"
Jul 28 10:23:55.036600 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:55.036572    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-8e237-predictor-8c874fbcc-fnrsp"]
Jul 28 10:23:55.038496 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:23:55.038469    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86f2d777_6f0c_445d_b9f1_d6b43e8ef110.slice/crio-f5f9476f844621d96abf42e26e5dc0a887af9abfa3434f153f45803d2150dcde WatchSource:0}: Error finding container f5f9476f844621d96abf42e26e5dc0a887af9abfa3434f153f45803d2150dcde: Status 404 returned error can't find the container with id f5f9476f844621d96abf42e26e5dc0a887af9abfa3434f153f45803d2150dcde
Jul 28 10:23:55.040916 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:55.040897    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 10:23:55.656471 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:55.656436    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-8e237-predictor-8c874fbcc-fnrsp" event={"ID":"86f2d777-6f0c-445d-b9f1-d6b43e8ef110","Type":"ContainerStarted","Data":"f5f9476f844621d96abf42e26e5dc0a887af9abfa3434f153f45803d2150dcde"}
Jul 28 10:23:56.192062 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:56.192026    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq" podUID="3ff171d2-0800-4761-aaf9-193a083ed88b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 28 10:23:56.662573 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:56.662542    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-8e237-predictor-8c874fbcc-fnrsp" event={"ID":"86f2d777-6f0c-445d-b9f1-d6b43e8ef110","Type":"ContainerStarted","Data":"3c756cebb30a7c49dce4c109bbb238bfe625f3dc4ef59c3d3cd9cb066cf568e1"}
Jul 28 10:23:56.662761 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:56.662598    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-8e237-predictor-8c874fbcc-fnrsp" event={"ID":"86f2d777-6f0c-445d-b9f1-d6b43e8ef110","Type":"ContainerStarted","Data":"957a41e275fd1096ba6e08b5257ac43a90484f87e5bee0b595b7407d2de27f03"}
Jul 28 10:23:56.662761 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:56.662687    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-8e237-predictor-8c874fbcc-fnrsp"
Jul 28 10:23:56.681312 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:56.681273    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-8e237-predictor-8c874fbcc-fnrsp" podStartSLOduration=2.51608337 podStartE2EDuration="3.681260388s" totalImagesPullingTime="1.165177018s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:23:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:23:56.679389944 +0000 UTC m=+975.779896499" watchObservedRunningTime="2026-07-28 10:23:56.681260388 +0000 UTC m=+975.781766941"
Jul 28 10:23:57.196720 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:57.196684    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt" podUID="3a48daa8-0991-4082-a471-629bb7afdf09" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jul 28 10:23:57.201042 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:57.201017    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt" podUID="3a48daa8-0991-4082-a471-629bb7afdf09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 10:23:57.380954 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:57.380930    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt"
Jul 28 10:23:57.484387 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:57.484322    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a48daa8-0991-4082-a471-629bb7afdf09-proxy-tls\") pod \"3a48daa8-0991-4082-a471-629bb7afdf09\" (UID: \"3a48daa8-0991-4082-a471-629bb7afdf09\") "
Jul 28 10:23:57.484387 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:57.484375    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a48daa8-0991-4082-a471-629bb7afdf09-kserve-provision-location\") pod \"3a48daa8-0991-4082-a471-629bb7afdf09\" (UID: \"3a48daa8-0991-4082-a471-629bb7afdf09\") "
Jul 28 10:23:57.484555 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:57.484396    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a48daa8-0991-4082-a471-629bb7afdf09-cabundle-cert\") pod \"3a48daa8-0991-4082-a471-629bb7afdf09\" (UID: \"3a48daa8-0991-4082-a471-629bb7afdf09\") "
Jul 28 10:23:57.484555 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:57.484439    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-drxx5\" (UniqueName: \"kubernetes.io/projected/3a48daa8-0991-4082-a471-629bb7afdf09-kube-api-access-drxx5\") pod \"3a48daa8-0991-4082-a471-629bb7afdf09\" (UID: \"3a48daa8-0991-4082-a471-629bb7afdf09\") "
Jul 28 10:23:57.484555 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:57.484459    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a48daa8-0991-4082-a471-629bb7afdf09-kube-rbac-proxy-sar-config\") pod \"3a48daa8-0991-4082-a471-629bb7afdf09\" (UID: \"3a48daa8-0991-4082-a471-629bb7afdf09\") "
Jul 28 10:23:57.484788 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:57.484758    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a48daa8-0991-4082-a471-629bb7afdf09-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3a48daa8-0991-4082-a471-629bb7afdf09" (UID: "3a48daa8-0991-4082-a471-629bb7afdf09"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:23:57.484849 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:57.484806    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a48daa8-0991-4082-a471-629bb7afdf09-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3a48daa8-0991-4082-a471-629bb7afdf09" (UID: "3a48daa8-0991-4082-a471-629bb7afdf09"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:23:57.484888 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:57.484855    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a48daa8-0991-4082-a471-629bb7afdf09-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3a48daa8-0991-4082-a471-629bb7afdf09" (UID: "3a48daa8-0991-4082-a471-629bb7afdf09"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:23:57.486393 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:57.486371    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a48daa8-0991-4082-a471-629bb7afdf09-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3a48daa8-0991-4082-a471-629bb7afdf09" (UID: "3a48daa8-0991-4082-a471-629bb7afdf09"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:23:57.486643 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:57.486622    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a48daa8-0991-4082-a471-629bb7afdf09-kube-api-access-drxx5" (OuterVolumeSpecName: "kube-api-access-drxx5") pod "3a48daa8-0991-4082-a471-629bb7afdf09" (UID: "3a48daa8-0991-4082-a471-629bb7afdf09"). InnerVolumeSpecName "kube-api-access-drxx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:23:57.585800 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:57.585778    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a48daa8-0991-4082-a471-629bb7afdf09-kserve-provision-location\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:23:57.585800 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:57.585797    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a48daa8-0991-4082-a471-629bb7afdf09-cabundle-cert\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:23:57.585927 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:57.585807    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drxx5\" (UniqueName: \"kubernetes.io/projected/3a48daa8-0991-4082-a471-629bb7afdf09-kube-api-access-drxx5\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:23:57.585927 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:57.585816    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a48daa8-0991-4082-a471-629bb7afdf09-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:23:57.585927 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:57.585824    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a48daa8-0991-4082-a471-629bb7afdf09-proxy-tls\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:23:57.668992 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:57.668962    2583 generic.go:356] "Generic (PLEG): container finished" podID="3a48daa8-0991-4082-a471-629bb7afdf09" containerID="cbb064ef3c18f48638fdee6960ee7e725147b9d71358ec39360efce929cc2b5e" exitCode=0
Jul 28 10:23:57.669080 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:57.669039    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt"
Jul 28 10:23:57.669080 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:57.669040    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt" event={"ID":"3a48daa8-0991-4082-a471-629bb7afdf09","Type":"ContainerDied","Data":"cbb064ef3c18f48638fdee6960ee7e725147b9d71358ec39360efce929cc2b5e"}
Jul 28 10:23:57.669158 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:57.669111    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt" event={"ID":"3a48daa8-0991-4082-a471-629bb7afdf09","Type":"ContainerDied","Data":"b790a5e8e42ec5b312ae7572064a958a5bfe305c51d2a4079e1f19d01e656d2c"}
Jul 28 10:23:57.669158 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:57.669142    2583 scope.go:117] "RemoveContainer" containerID="4ab09614b0452242a1f574cefc9eca19a13e75e7ec98028e79346655d14ca0d0"
Jul 28 10:23:57.669618 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:57.669599    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-8e237-predictor-8c874fbcc-fnrsp"
Jul 28 10:23:57.671099 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:57.671079    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-8e237-predictor-8c874fbcc-fnrsp"
Jul 28 10:23:57.677647 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:57.677633    2583 scope.go:117] "RemoveContainer" containerID="cbb064ef3c18f48638fdee6960ee7e725147b9d71358ec39360efce929cc2b5e"
Jul 28 10:23:57.684889 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:57.684874    2583 scope.go:117] "RemoveContainer" containerID="748ac6aaaf1b2a1fa052440ab56f31eaa251c49e04eda0df5a8338ed0b429298"
Jul 28 10:23:57.691915 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:57.691899    2583 scope.go:117] "RemoveContainer" containerID="4ab09614b0452242a1f574cefc9eca19a13e75e7ec98028e79346655d14ca0d0"
Jul 28 10:23:57.692159 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:23:57.692140    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ab09614b0452242a1f574cefc9eca19a13e75e7ec98028e79346655d14ca0d0\": container with ID starting with 4ab09614b0452242a1f574cefc9eca19a13e75e7ec98028e79346655d14ca0d0 not found: ID does not exist" containerID="4ab09614b0452242a1f574cefc9eca19a13e75e7ec98028e79346655d14ca0d0"
Jul 28 10:23:57.692225 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:57.692167    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ab09614b0452242a1f574cefc9eca19a13e75e7ec98028e79346655d14ca0d0"} err="failed to get container status \"4ab09614b0452242a1f574cefc9eca19a13e75e7ec98028e79346655d14ca0d0\": rpc error: code = NotFound desc = could not find container \"4ab09614b0452242a1f574cefc9eca19a13e75e7ec98028e79346655d14ca0d0\": container with ID starting with 4ab09614b0452242a1f574cefc9eca19a13e75e7ec98028e79346655d14ca0d0 not found: ID does not exist"
Jul 28 10:23:57.692225 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:57.692208    2583 scope.go:117] "RemoveContainer" containerID="cbb064ef3c18f48638fdee6960ee7e725147b9d71358ec39360efce929cc2b5e"
Jul 28 10:23:57.692463 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:23:57.692445    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbb064ef3c18f48638fdee6960ee7e725147b9d71358ec39360efce929cc2b5e\": container with ID starting with cbb064ef3c18f48638fdee6960ee7e725147b9d71358ec39360efce929cc2b5e not found: ID does not exist" containerID="cbb064ef3c18f48638fdee6960ee7e725147b9d71358ec39360efce929cc2b5e"
Jul 28 10:23:57.692526 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:57.692472    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbb064ef3c18f48638fdee6960ee7e725147b9d71358ec39360efce929cc2b5e"} err="failed to get container status \"cbb064ef3c18f48638fdee6960ee7e725147b9d71358ec39360efce929cc2b5e\": rpc error: code = NotFound desc = could not find container \"cbb064ef3c18f48638fdee6960ee7e725147b9d71358ec39360efce929cc2b5e\": container with ID starting with cbb064ef3c18f48638fdee6960ee7e725147b9d71358ec39360efce929cc2b5e not found: ID does not exist"
Jul 28 10:23:57.692526 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:57.692494    2583 scope.go:117] "RemoveContainer" containerID="748ac6aaaf1b2a1fa052440ab56f31eaa251c49e04eda0df5a8338ed0b429298"
Jul 28 10:23:57.692703 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:23:57.692687    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"748ac6aaaf1b2a1fa052440ab56f31eaa251c49e04eda0df5a8338ed0b429298\": container with ID starting with 748ac6aaaf1b2a1fa052440ab56f31eaa251c49e04eda0df5a8338ed0b429298 not found: ID does not exist" containerID="748ac6aaaf1b2a1fa052440ab56f31eaa251c49e04eda0df5a8338ed0b429298"
Jul 28 10:23:57.692741 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:57.692707    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"748ac6aaaf1b2a1fa052440ab56f31eaa251c49e04eda0df5a8338ed0b429298"} err="failed to get container status \"748ac6aaaf1b2a1fa052440ab56f31eaa251c49e04eda0df5a8338ed0b429298\": rpc error: code = NotFound desc = could not find container \"748ac6aaaf1b2a1fa052440ab56f31eaa251c49e04eda0df5a8338ed0b429298\": container with ID starting with 748ac6aaaf1b2a1fa052440ab56f31eaa251c49e04eda0df5a8338ed0b429298 not found: ID does not exist"
Jul 28 10:23:57.704875 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:57.704856    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt"]
Jul 28 10:23:57.711822 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:57.711803    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-b0501-predictor-696db4b46f-d7ggt"]
Jul 28 10:23:58.470665 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:58.470644    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq"
Jul 28 10:23:58.493161 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:58.493111    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ff171d2-0800-4761-aaf9-193a083ed88b-proxy-tls\") pod \"3ff171d2-0800-4761-aaf9-193a083ed88b\" (UID: \"3ff171d2-0800-4761-aaf9-193a083ed88b\") "
Jul 28 10:23:58.493287 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:58.493164    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3ff171d2-0800-4761-aaf9-193a083ed88b-cabundle-cert\") pod \"3ff171d2-0800-4761-aaf9-193a083ed88b\" (UID: \"3ff171d2-0800-4761-aaf9-193a083ed88b\") "
Jul 28 10:23:58.493287 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:58.493198    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5q8mg\" (UniqueName: \"kubernetes.io/projected/3ff171d2-0800-4761-aaf9-193a083ed88b-kube-api-access-5q8mg\") pod \"3ff171d2-0800-4761-aaf9-193a083ed88b\" (UID: \"3ff171d2-0800-4761-aaf9-193a083ed88b\") "
Jul 28 10:23:58.493287 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:58.493244    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3ff171d2-0800-4761-aaf9-193a083ed88b-kserve-provision-location\") pod \"3ff171d2-0800-4761-aaf9-193a083ed88b\" (UID: \"3ff171d2-0800-4761-aaf9-193a083ed88b\") "
Jul 28 10:23:58.493467 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:58.493298    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ff171d2-0800-4761-aaf9-193a083ed88b-kube-rbac-proxy-sar-config\") pod \"3ff171d2-0800-4761-aaf9-193a083ed88b\" (UID: \"3ff171d2-0800-4761-aaf9-193a083ed88b\") "
Jul 28 10:23:58.493548 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:58.493522    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ff171d2-0800-4761-aaf9-193a083ed88b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3ff171d2-0800-4761-aaf9-193a083ed88b" (UID: "3ff171d2-0800-4761-aaf9-193a083ed88b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:23:58.493612 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:58.493579    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ff171d2-0800-4761-aaf9-193a083ed88b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3ff171d2-0800-4761-aaf9-193a083ed88b" (UID: "3ff171d2-0800-4761-aaf9-193a083ed88b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:23:58.493838 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:58.493807    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ff171d2-0800-4761-aaf9-193a083ed88b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3ff171d2-0800-4761-aaf9-193a083ed88b" (UID: "3ff171d2-0800-4761-aaf9-193a083ed88b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:23:58.495240 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:58.495174    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ff171d2-0800-4761-aaf9-193a083ed88b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3ff171d2-0800-4761-aaf9-193a083ed88b" (UID: "3ff171d2-0800-4761-aaf9-193a083ed88b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:23:58.495442 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:58.495420    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ff171d2-0800-4761-aaf9-193a083ed88b-kube-api-access-5q8mg" (OuterVolumeSpecName: "kube-api-access-5q8mg") pod "3ff171d2-0800-4761-aaf9-193a083ed88b" (UID: "3ff171d2-0800-4761-aaf9-193a083ed88b"). InnerVolumeSpecName "kube-api-access-5q8mg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:23:58.594114 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:58.594095    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ff171d2-0800-4761-aaf9-193a083ed88b-proxy-tls\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:23:58.594114 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:58.594115    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3ff171d2-0800-4761-aaf9-193a083ed88b-cabundle-cert\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:23:58.594114 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:58.594123    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5q8mg\" (UniqueName: \"kubernetes.io/projected/3ff171d2-0800-4761-aaf9-193a083ed88b-kube-api-access-5q8mg\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:23:58.594300 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:58.594132    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3ff171d2-0800-4761-aaf9-193a083ed88b-kserve-provision-location\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:23:58.594300 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:58.594142    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ff171d2-0800-4761-aaf9-193a083ed88b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:23:58.675055 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:58.675030    2583 generic.go:356] "Generic (PLEG): container finished" podID="3ff171d2-0800-4761-aaf9-193a083ed88b" containerID="7a4cab835439f808773d7e28725d98dd03df9236363739eae82d1fe40427b958" exitCode=0
Jul 28 10:23:58.675143 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:58.675086    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq"
Jul 28 10:23:58.675143 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:58.675114    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq" event={"ID":"3ff171d2-0800-4761-aaf9-193a083ed88b","Type":"ContainerDied","Data":"7a4cab835439f808773d7e28725d98dd03df9236363739eae82d1fe40427b958"}
Jul 28 10:23:58.675273 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:58.675222    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq" event={"ID":"3ff171d2-0800-4761-aaf9-193a083ed88b","Type":"ContainerDied","Data":"3e7eaf23de02491455988e3b7781b56939380c5485f9ec82341d3c9195e3f916"}
Jul 28 10:23:58.675273 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:58.675266    2583 scope.go:117] "RemoveContainer" containerID="37a7f64c03a3bd2bb2e08debff56aaa172ae8ac5c1b36f77cf9bbdc0a15e8576"
Jul 28 10:23:58.683408 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:58.683393    2583 scope.go:117] "RemoveContainer" containerID="7a4cab835439f808773d7e28725d98dd03df9236363739eae82d1fe40427b958"
Jul 28 10:23:58.690675 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:58.690659    2583 scope.go:117] "RemoveContainer" containerID="5a7c0a84fe9fa3c520b471e932e0a44c89742597b9cb6e7feb4fff343478da74"
Jul 28 10:23:58.697893 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:58.697876    2583 scope.go:117] "RemoveContainer" containerID="37a7f64c03a3bd2bb2e08debff56aaa172ae8ac5c1b36f77cf9bbdc0a15e8576"
Jul 28 10:23:58.698133 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:23:58.698113    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37a7f64c03a3bd2bb2e08debff56aaa172ae8ac5c1b36f77cf9bbdc0a15e8576\": container with ID starting with 37a7f64c03a3bd2bb2e08debff56aaa172ae8ac5c1b36f77cf9bbdc0a15e8576 not found: ID does not exist" containerID="37a7f64c03a3bd2bb2e08debff56aaa172ae8ac5c1b36f77cf9bbdc0a15e8576"
Jul 28 10:23:58.698236 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:58.698136    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37a7f64c03a3bd2bb2e08debff56aaa172ae8ac5c1b36f77cf9bbdc0a15e8576"} err="failed to get container status \"37a7f64c03a3bd2bb2e08debff56aaa172ae8ac5c1b36f77cf9bbdc0a15e8576\": rpc error: code = NotFound desc = could not find container \"37a7f64c03a3bd2bb2e08debff56aaa172ae8ac5c1b36f77cf9bbdc0a15e8576\": container with ID starting with 37a7f64c03a3bd2bb2e08debff56aaa172ae8ac5c1b36f77cf9bbdc0a15e8576 not found: ID does not exist"
Jul 28 10:23:58.698236 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:58.698152    2583 scope.go:117] "RemoveContainer" containerID="7a4cab835439f808773d7e28725d98dd03df9236363739eae82d1fe40427b958"
Jul 28 10:23:58.698425 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:23:58.698406    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a4cab835439f808773d7e28725d98dd03df9236363739eae82d1fe40427b958\": container with ID starting with 7a4cab835439f808773d7e28725d98dd03df9236363739eae82d1fe40427b958 not found: ID does not exist" containerID="7a4cab835439f808773d7e28725d98dd03df9236363739eae82d1fe40427b958"
Jul 28 10:23:58.698471 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:58.698431    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a4cab835439f808773d7e28725d98dd03df9236363739eae82d1fe40427b958"} err="failed to get container status \"7a4cab835439f808773d7e28725d98dd03df9236363739eae82d1fe40427b958\": rpc error: code = NotFound desc = could not find container \"7a4cab835439f808773d7e28725d98dd03df9236363739eae82d1fe40427b958\": container with ID starting with 7a4cab835439f808773d7e28725d98dd03df9236363739eae82d1fe40427b958 not found: ID does not exist"
Jul 28 10:23:58.698471 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:58.698445    2583 scope.go:117] "RemoveContainer" containerID="5a7c0a84fe9fa3c520b471e932e0a44c89742597b9cb6e7feb4fff343478da74"
Jul 28 10:23:58.698685 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:23:58.698668    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a7c0a84fe9fa3c520b471e932e0a44c89742597b9cb6e7feb4fff343478da74\": container with ID starting with 5a7c0a84fe9fa3c520b471e932e0a44c89742597b9cb6e7feb4fff343478da74 not found: ID does not exist" containerID="5a7c0a84fe9fa3c520b471e932e0a44c89742597b9cb6e7feb4fff343478da74"
Jul 28 10:23:58.698737 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:58.698697    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a7c0a84fe9fa3c520b471e932e0a44c89742597b9cb6e7feb4fff343478da74"} err="failed to get container status \"5a7c0a84fe9fa3c520b471e932e0a44c89742597b9cb6e7feb4fff343478da74\": rpc error: code = NotFound desc = could not find container \"5a7c0a84fe9fa3c520b471e932e0a44c89742597b9cb6e7feb4fff343478da74\": container with ID starting with 5a7c0a84fe9fa3c520b471e932e0a44c89742597b9cb6e7feb4fff343478da74 not found: ID does not exist"
Jul 28 10:23:58.698894 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:58.698876    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq"]
Jul 28 10:23:58.703487 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:58.703468    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-b0501-predictor-549c8f788f-ss4wq"]
Jul 28 10:23:59.438500 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:59.438467    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a48daa8-0991-4082-a471-629bb7afdf09" path="/var/lib/kubelet/pods/3a48daa8-0991-4082-a471-629bb7afdf09/volumes"
Jul 28 10:23:59.439002 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:23:59.438985    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ff171d2-0800-4761-aaf9-193a083ed88b" path="/var/lib/kubelet/pods/3ff171d2-0800-4761-aaf9-193a083ed88b/volumes"
Jul 28 10:24:04.687056 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:24:04.686971    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-8e237-predictor-8c874fbcc-fnrsp"
Jul 28 10:24:13.959795 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:24:13.959759    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:24:13.992559 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:24:13.992534    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht"]
Jul 28 10:24:13.992927 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:24:13.992900    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a48daa8-0991-4082-a471-629bb7afdf09" containerName="kube-rbac-proxy"
Jul 28 10:24:13.992927 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:24:13.992917    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a48daa8-0991-4082-a471-629bb7afdf09" containerName="kube-rbac-proxy"
Jul 28 10:24:13.993101 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:24:13.992939    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a48daa8-0991-4082-a471-629bb7afdf09" containerName="kserve-container"
Jul 28 10:24:13.993101 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:24:13.992947    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a48daa8-0991-4082-a471-629bb7afdf09" containerName="kserve-container"
Jul 28 10:24:13.993101 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:24:13.992959    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a48daa8-0991-4082-a471-629bb7afdf09" containerName="storage-initializer"
Jul 28 10:24:13.993101 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:24:13.992967    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a48daa8-0991-4082-a471-629bb7afdf09" containerName="storage-initializer"
Jul 28 10:24:13.993101 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:24:13.992986    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ff171d2-0800-4761-aaf9-193a083ed88b" containerName="kube-rbac-proxy"
Jul 28 10:24:13.993101 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:24:13.992994    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ff171d2-0800-4761-aaf9-193a083ed88b" containerName="kube-rbac-proxy"
Jul 28 10:24:13.993101 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:24:13.993006    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ff171d2-0800-4761-aaf9-193a083ed88b" containerName="kserve-container"
Jul 28 10:24:13.993101 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:24:13.993013    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ff171d2-0800-4761-aaf9-193a083ed88b" containerName="kserve-container"
Jul 28 10:24:13.993101 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:24:13.993027    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ff171d2-0800-4761-aaf9-193a083ed88b" containerName="storage-initializer"
Jul 28 10:24:13.993101 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:24:13.993035    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ff171d2-0800-4761-aaf9-193a083ed88b" containerName="storage-initializer"
Jul 28 10:24:13.993611 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:24:13.993127    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ff171d2-0800-4761-aaf9-193a083ed88b" containerName="kserve-container"
Jul 28 10:24:13.993611 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:24:13.993142    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a48daa8-0991-4082-a471-629bb7afdf09" containerName="kube-rbac-proxy"
Jul 28 10:24:13.993611 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:24:13.993152    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ff171d2-0800-4761-aaf9-193a083ed88b" containerName="kube-rbac-proxy"
Jul 28 10:24:13.993611 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:24:13.993162    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a48daa8-0991-4082-a471-629bb7afdf09" containerName="kserve-container"
Jul 28 10:24:14.000400 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:24:14.000378    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht"
Jul 28 10:24:14.002774 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:24:14.002749    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 10:24:14.002877 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:24:14.002756    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-8e237-predictor-serving-cert\""
Jul 28 10:24:14.003560 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:24:14.003068    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-8e237-kube-rbac-proxy-sar-config\""
Jul 28 10:24:14.003560 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:24:14.003345    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 10:24:14.006564 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:24:14.006547    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht"]
Jul 28 10:24:14.102537 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:24:14.102511    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d96f7a4-d01d-4770-9a32-878cd03cf875-proxy-tls\") pod \"isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht\" (UID: \"0d96f7a4-d01d-4770-9a32-878cd03cf875\") " pod="kserve-ci-e2e-test/isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht"
Jul 28 10:24:14.102634 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:24:14.102564    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvwl2\" (UniqueName: \"kubernetes.io/projected/0d96f7a4-d01d-4770-9a32-878cd03cf875-kube-api-access-tvwl2\") pod \"isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht\" (UID: \"0d96f7a4-d01d-4770-9a32-878cd03cf875\") " pod="kserve-ci-e2e-test/isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht"
Jul 28 10:24:14.102634 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:24:14.102620    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d96f7a4-d01d-4770-9a32-878cd03cf875-kserve-provision-location\") pod \"isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht\" (UID: \"0d96f7a4-d01d-4770-9a32-878cd03cf875\") " pod="kserve-ci-e2e-test/isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht"
Jul 28 10:24:14.102713 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:24:14.102644    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d96f7a4-d01d-4770-9a32-878cd03cf875-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht\" (UID: \"0d96f7a4-d01d-4770-9a32-878cd03cf875\") " pod="kserve-ci-e2e-test/isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht"
Jul 28 10:24:14.102713 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:24:14.102686    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d96f7a4-d01d-4770-9a32-878cd03cf875-cabundle-cert\") pod \"isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht\" (UID: \"0d96f7a4-d01d-4770-9a32-878cd03cf875\") " pod="kserve-ci-e2e-test/isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht"
Jul 28 10:24:14.203943 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:24:14.203918    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvwl2\" (UniqueName: \"kubernetes.io/projected/0d96f7a4-d01d-4770-9a32-878cd03cf875-kube-api-access-tvwl2\") pod \"isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht\" (UID: \"0d96f7a4-d01d-4770-9a32-878cd03cf875\") " pod="kserve-ci-e2e-test/isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht"
Jul 28 10:24:14.204039 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:24:14.203955    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d96f7a4-d01d-4770-9a32-878cd03cf875-kserve-provision-location\") pod \"isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht\" (UID: \"0d96f7a4-d01d-4770-9a32-878cd03cf875\") " pod="kserve-ci-e2e-test/isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht"
Jul 28 10:24:14.204039 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:24:14.203973    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d96f7a4-d01d-4770-9a32-878cd03cf875-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht\" (UID: \"0d96f7a4-d01d-4770-9a32-878cd03cf875\") " pod="kserve-ci-e2e-test/isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht"
Jul 28 10:24:14.204134 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:24:14.204118    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d96f7a4-d01d-4770-9a32-878cd03cf875-cabundle-cert\") pod \"isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht\" (UID: \"0d96f7a4-d01d-4770-9a32-878cd03cf875\") " pod="kserve-ci-e2e-test/isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht"
Jul 28 10:24:14.204231 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:24:14.204215    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d96f7a4-d01d-4770-9a32-878cd03cf875-proxy-tls\") pod \"isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht\" (UID: \"0d96f7a4-d01d-4770-9a32-878cd03cf875\") " pod="kserve-ci-e2e-test/isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht"
Jul 28 10:24:14.204410 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:24:14.204390    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d96f7a4-d01d-4770-9a32-878cd03cf875-kserve-provision-location\") pod \"isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht\" (UID: \"0d96f7a4-d01d-4770-9a32-878cd03cf875\") " pod="kserve-ci-e2e-test/isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht"
Jul 28 10:24:14.204636 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:24:14.204613    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d96f7a4-d01d-4770-9a32-878cd03cf875-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht\" (UID: \"0d96f7a4-d01d-4770-9a32-878cd03cf875\") " pod="kserve-ci-e2e-test/isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht"
Jul 28 10:24:14.204727 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:24:14.204673    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d96f7a4-d01d-4770-9a32-878cd03cf875-cabundle-cert\") pod \"isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht\" (UID: \"0d96f7a4-d01d-4770-9a32-878cd03cf875\") " pod="kserve-ci-e2e-test/isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht"
Jul 28 10:24:14.206494 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:24:14.206475    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d96f7a4-d01d-4770-9a32-878cd03cf875-proxy-tls\") pod \"isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht\" (UID: \"0d96f7a4-d01d-4770-9a32-878cd03cf875\") " pod="kserve-ci-e2e-test/isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht"
Jul 28 10:24:14.212756 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:24:14.212710    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvwl2\" (UniqueName: \"kubernetes.io/projected/0d96f7a4-d01d-4770-9a32-878cd03cf875-kube-api-access-tvwl2\") pod \"isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht\" (UID: \"0d96f7a4-d01d-4770-9a32-878cd03cf875\") " pod="kserve-ci-e2e-test/isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht"
Jul 28 10:24:14.312784 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:24:14.312761    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht"
Jul 28 10:24:14.449508 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:24:14.449484    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht"]
Jul 28 10:24:14.451108 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:24:14.451083    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d96f7a4_d01d_4770_9a32_878cd03cf875.slice/crio-0c9b220e8eb7c732ceef82daf03cc4b0e03463ff02e30ff4c01a1149049659eb WatchSource:0}: Error finding container 0c9b220e8eb7c732ceef82daf03cc4b0e03463ff02e30ff4c01a1149049659eb: Status 404 returned error can't find the container with id 0c9b220e8eb7c732ceef82daf03cc4b0e03463ff02e30ff4c01a1149049659eb
Jul 28 10:24:14.752311 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:24:14.752236    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht" event={"ID":"0d96f7a4-d01d-4770-9a32-878cd03cf875","Type":"ContainerStarted","Data":"a2b42f8068fd96683b80c1e2f0d3cdb1f71b19dae7ac5cdae503130aadae1fbf"}
Jul 28 10:24:14.752619 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:24:14.752327    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht" event={"ID":"0d96f7a4-d01d-4770-9a32-878cd03cf875","Type":"ContainerStarted","Data":"0c9b220e8eb7c732ceef82daf03cc4b0e03463ff02e30ff4c01a1149049659eb"}
Jul 28 10:24:14.964314 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:24:14.964287    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:24:18.773474 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:24:18.773389    2583 generic.go:356] "Generic (PLEG): container finished" podID="0d96f7a4-d01d-4770-9a32-878cd03cf875" containerID="a2b42f8068fd96683b80c1e2f0d3cdb1f71b19dae7ac5cdae503130aadae1fbf" exitCode=0
Jul 28 10:24:18.773825 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:24:18.773467    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht" event={"ID":"0d96f7a4-d01d-4770-9a32-878cd03cf875","Type":"ContainerDied","Data":"a2b42f8068fd96683b80c1e2f0d3cdb1f71b19dae7ac5cdae503130aadae1fbf"}
Jul 28 10:24:19.783666 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:24:19.783630    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht" event={"ID":"0d96f7a4-d01d-4770-9a32-878cd03cf875","Type":"ContainerStarted","Data":"b1650c28b434981ab211532d0b7fd81aa5116df026e8145378197ee0c1f9b632"}
Jul 28 10:24:19.784029 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:24:19.783698    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht" event={"ID":"0d96f7a4-d01d-4770-9a32-878cd03cf875","Type":"ContainerStarted","Data":"6ad42a0ccaa46289c4ab56cc95c7c0d63c1c5b854d312d5287bf1adb2454fe66"}
Jul 28 10:24:19.784029 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:24:19.783731    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht" event={"ID":"0d96f7a4-d01d-4770-9a32-878cd03cf875","Type":"ContainerStarted","Data":"fc6ecdef06e73643c30149e22425c72c6031117d401bfe79389558b25a00084e"}
Jul 28 10:24:19.784029 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:24:19.784013    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht"
Jul 28 10:24:19.784220 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:24:19.784206    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht"
Jul 28 10:24:19.785284 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:24:19.785263    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht" podUID="0d96f7a4-d01d-4770-9a32-878cd03cf875" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 10:24:19.811088 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:24:19.806960    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht" podStartSLOduration=2.912801274 podStartE2EDuration="6.806944312s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.894143038s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:24:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:24:19.803734085 +0000 UTC m=+998.904240638" watchObservedRunningTime="2026-07-28 10:24:19.806944312 +0000 UTC m=+998.907450870"
Jul 28 10:24:20.788519 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:24:20.788457    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht" podUID="0d96f7a4-d01d-4770-9a32-878cd03cf875" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 10:24:20.788519 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:24:20.788490    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht"
Jul 28 10:24:20.789752 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:24:20.789723    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht" podUID="0d96f7a4-d01d-4770-9a32-878cd03cf875" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 10:24:21.793689 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:24:21.793646    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht" podUID="0d96f7a4-d01d-4770-9a32-878cd03cf875" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 10:24:21.794089 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:24:21.793955    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht" podUID="0d96f7a4-d01d-4770-9a32-878cd03cf875" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 10:24:26.797635 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:24:26.797602    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht"
Jul 28 10:24:26.798281 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:24:26.798247    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht" podUID="0d96f7a4-d01d-4770-9a32-878cd03cf875" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 10:24:26.798651 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:24:26.798612    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht" podUID="0d96f7a4-d01d-4770-9a32-878cd03cf875" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 10:24:36.799067 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:24:36.799026    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht" podUID="0d96f7a4-d01d-4770-9a32-878cd03cf875" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 10:24:36.799590 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:24:36.799562    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht" podUID="0d96f7a4-d01d-4770-9a32-878cd03cf875" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 10:24:46.798756 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:24:46.798704    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht" podUID="0d96f7a4-d01d-4770-9a32-878cd03cf875" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 10:24:46.799306 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:24:46.799152    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht" podUID="0d96f7a4-d01d-4770-9a32-878cd03cf875" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 10:24:53.897241 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:24:53.897201    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:24:53.954159 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:24:53.954129    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:24:56.798628 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:24:56.798581    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht" podUID="0d96f7a4-d01d-4770-9a32-878cd03cf875" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 10:24:56.799247 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:24:56.799206    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht" podUID="0d96f7a4-d01d-4770-9a32-878cd03cf875" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 10:25:06.798550 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:06.798513    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht" podUID="0d96f7a4-d01d-4770-9a32-878cd03cf875" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 10:25:06.799060 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:06.799037    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht" podUID="0d96f7a4-d01d-4770-9a32-878cd03cf875" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 10:25:16.798590 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:16.798553    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht" podUID="0d96f7a4-d01d-4770-9a32-878cd03cf875" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 10:25:16.799010 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:16.798988    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht" podUID="0d96f7a4-d01d-4770-9a32-878cd03cf875" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 10:25:26.799363 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:26.799332    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht"
Jul 28 10:25:26.799898 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:26.799448    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht"
Jul 28 10:25:39.003379 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:39.003280    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-8e237-predictor-8c874fbcc-fnrsp_86f2d777-6f0c-445d-b9f1-d6b43e8ef110/kserve-container/0.log"
Jul 28 10:25:39.179959 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:39.179925    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht"]
Jul 28 10:25:39.180346 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:39.180289    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht" podUID="0d96f7a4-d01d-4770-9a32-878cd03cf875" containerName="kserve-container" containerID="cri-o://fc6ecdef06e73643c30149e22425c72c6031117d401bfe79389558b25a00084e" gracePeriod=30
Jul 28 10:25:39.180579 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:39.180399    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht" podUID="0d96f7a4-d01d-4770-9a32-878cd03cf875" containerName="agent" containerID="cri-o://b1650c28b434981ab211532d0b7fd81aa5116df026e8145378197ee0c1f9b632" gracePeriod=30
Jul 28 10:25:39.180708 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:39.180395    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht" podUID="0d96f7a4-d01d-4770-9a32-878cd03cf875" containerName="kube-rbac-proxy" containerID="cri-o://6ad42a0ccaa46289c4ab56cc95c7c0d63c1c5b854d312d5287bf1adb2454fe66" gracePeriod=30
Jul 28 10:25:39.187905 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:39.187877    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:25:39.233473 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:39.233436    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-8e237-predictor-8c874fbcc-fnrsp"]
Jul 28 10:25:39.233711 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:39.233687    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-8e237-predictor-8c874fbcc-fnrsp" podUID="86f2d777-6f0c-445d-b9f1-d6b43e8ef110" containerName="kserve-container" containerID="cri-o://957a41e275fd1096ba6e08b5257ac43a90484f87e5bee0b595b7407d2de27f03" gracePeriod=30
Jul 28 10:25:39.233811 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:39.233783    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-8e237-predictor-8c874fbcc-fnrsp" podUID="86f2d777-6f0c-445d-b9f1-d6b43e8ef110" containerName="kube-rbac-proxy" containerID="cri-o://3c756cebb30a7c49dce4c109bbb238bfe625f3dc4ef59c3d3cd9cb066cf568e1" gracePeriod=30
Jul 28 10:25:39.262075 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:39.262017    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:25:39.262508 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:39.262488    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455"]
Jul 28 10:25:39.266343 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:39.266325    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455"
Jul 28 10:25:39.268277 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:39.268254    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-d2f58-predictor-serving-cert\""
Jul 28 10:25:39.268373 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:39.268348    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-d2f58-kube-rbac-proxy-sar-config\""
Jul 28 10:25:39.276443 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:39.276418    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455"]
Jul 28 10:25:39.313875 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:39.313852    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:25:39.332966 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:39.332947    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a58f731f-e46a-4cc4-b088-7dc00d15bb40-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455\" (UID: \"a58f731f-e46a-4cc4-b088-7dc00d15bb40\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455"
Jul 28 10:25:39.333082 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:39.332978    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a58f731f-e46a-4cc4-b088-7dc00d15bb40-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455\" (UID: \"a58f731f-e46a-4cc4-b088-7dc00d15bb40\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455"
Jul 28 10:25:39.333082 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:39.332999    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nj9c\" (UniqueName: \"kubernetes.io/projected/a58f731f-e46a-4cc4-b088-7dc00d15bb40-kube-api-access-4nj9c\") pod \"isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455\" (UID: \"a58f731f-e46a-4cc4-b088-7dc00d15bb40\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455"
Jul 28 10:25:39.333082 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:39.333015    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a58f731f-e46a-4cc4-b088-7dc00d15bb40-cabundle-cert\") pod \"isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455\" (UID: \"a58f731f-e46a-4cc4-b088-7dc00d15bb40\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455"
Jul 28 10:25:39.333269 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:39.333114    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a58f731f-e46a-4cc4-b088-7dc00d15bb40-proxy-tls\") pod \"isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455\" (UID: \"a58f731f-e46a-4cc4-b088-7dc00d15bb40\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455"
Jul 28 10:25:39.433583 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:39.433548    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a58f731f-e46a-4cc4-b088-7dc00d15bb40-proxy-tls\") pod \"isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455\" (UID: \"a58f731f-e46a-4cc4-b088-7dc00d15bb40\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455"
Jul 28 10:25:39.433740 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:39.433631    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a58f731f-e46a-4cc4-b088-7dc00d15bb40-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455\" (UID: \"a58f731f-e46a-4cc4-b088-7dc00d15bb40\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455"
Jul 28 10:25:39.433740 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:25:39.433717    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-scale-raw-d2f58-predictor-serving-cert: secret "isvc-sklearn-scale-raw-d2f58-predictor-serving-cert" not found
Jul 28 10:25:39.433856 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:25:39.433789    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a58f731f-e46a-4cc4-b088-7dc00d15bb40-proxy-tls podName:a58f731f-e46a-4cc4-b088-7dc00d15bb40 nodeName:}" failed. No retries permitted until 2026-07-28 10:25:39.933769307 +0000 UTC m=+1079.034275840 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a58f731f-e46a-4cc4-b088-7dc00d15bb40-proxy-tls") pod "isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455" (UID: "a58f731f-e46a-4cc4-b088-7dc00d15bb40") : secret "isvc-sklearn-scale-raw-d2f58-predictor-serving-cert" not found
Jul 28 10:25:39.433930 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:39.433911    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a58f731f-e46a-4cc4-b088-7dc00d15bb40-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455\" (UID: \"a58f731f-e46a-4cc4-b088-7dc00d15bb40\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455"
Jul 28 10:25:39.433987 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:39.433952    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nj9c\" (UniqueName: \"kubernetes.io/projected/a58f731f-e46a-4cc4-b088-7dc00d15bb40-kube-api-access-4nj9c\") pod \"isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455\" (UID: \"a58f731f-e46a-4cc4-b088-7dc00d15bb40\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455"
Jul 28 10:25:39.433987 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:39.433976    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a58f731f-e46a-4cc4-b088-7dc00d15bb40-cabundle-cert\") pod \"isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455\" (UID: \"a58f731f-e46a-4cc4-b088-7dc00d15bb40\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455"
Jul 28 10:25:39.434384 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:39.434359    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a58f731f-e46a-4cc4-b088-7dc00d15bb40-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455\" (UID: \"a58f731f-e46a-4cc4-b088-7dc00d15bb40\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455"
Jul 28 10:25:39.434500 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:39.434382    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a58f731f-e46a-4cc4-b088-7dc00d15bb40-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455\" (UID: \"a58f731f-e46a-4cc4-b088-7dc00d15bb40\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455"
Jul 28 10:25:39.434769 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:39.434748    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a58f731f-e46a-4cc4-b088-7dc00d15bb40-cabundle-cert\") pod \"isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455\" (UID: \"a58f731f-e46a-4cc4-b088-7dc00d15bb40\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455"
Jul 28 10:25:39.442359 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:39.442337    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nj9c\" (UniqueName: \"kubernetes.io/projected/a58f731f-e46a-4cc4-b088-7dc00d15bb40-kube-api-access-4nj9c\") pod \"isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455\" (UID: \"a58f731f-e46a-4cc4-b088-7dc00d15bb40\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455"
Jul 28 10:25:39.487286 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:39.487266    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-8e237-predictor-8c874fbcc-fnrsp"
Jul 28 10:25:39.534415 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:39.534331    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86f2d777-6f0c-445d-b9f1-d6b43e8ef110-proxy-tls\") pod \"86f2d777-6f0c-445d-b9f1-d6b43e8ef110\" (UID: \"86f2d777-6f0c-445d-b9f1-d6b43e8ef110\") "
Jul 28 10:25:39.534415 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:39.534400    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2ccnz\" (UniqueName: \"kubernetes.io/projected/86f2d777-6f0c-445d-b9f1-d6b43e8ef110-kube-api-access-2ccnz\") pod \"86f2d777-6f0c-445d-b9f1-d6b43e8ef110\" (UID: \"86f2d777-6f0c-445d-b9f1-d6b43e8ef110\") "
Jul 28 10:25:39.534620 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:39.534481    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86f2d777-6f0c-445d-b9f1-d6b43e8ef110-kube-rbac-proxy-sar-config\") pod \"86f2d777-6f0c-445d-b9f1-d6b43e8ef110\" (UID: \"86f2d777-6f0c-445d-b9f1-d6b43e8ef110\") "
Jul 28 10:25:39.534824 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:39.534799    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86f2d777-6f0c-445d-b9f1-d6b43e8ef110-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "86f2d777-6f0c-445d-b9f1-d6b43e8ef110" (UID: "86f2d777-6f0c-445d-b9f1-d6b43e8ef110"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:25:39.536438 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:39.536416    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86f2d777-6f0c-445d-b9f1-d6b43e8ef110-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "86f2d777-6f0c-445d-b9f1-d6b43e8ef110" (UID: "86f2d777-6f0c-445d-b9f1-d6b43e8ef110"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:25:39.536604 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:39.536585    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86f2d777-6f0c-445d-b9f1-d6b43e8ef110-kube-api-access-2ccnz" (OuterVolumeSpecName: "kube-api-access-2ccnz") pod "86f2d777-6f0c-445d-b9f1-d6b43e8ef110" (UID: "86f2d777-6f0c-445d-b9f1-d6b43e8ef110"). InnerVolumeSpecName "kube-api-access-2ccnz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:25:39.585447 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:39.585415    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:25:39.635741 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:39.635692    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86f2d777-6f0c-445d-b9f1-d6b43e8ef110-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:25:39.635741 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:39.635728    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86f2d777-6f0c-445d-b9f1-d6b43e8ef110-proxy-tls\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:25:39.635741 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:39.635738    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ccnz\" (UniqueName: \"kubernetes.io/projected/86f2d777-6f0c-445d-b9f1-d6b43e8ef110-kube-api-access-2ccnz\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:25:39.938035 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:39.937996    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a58f731f-e46a-4cc4-b088-7dc00d15bb40-proxy-tls\") pod \"isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455\" (UID: \"a58f731f-e46a-4cc4-b088-7dc00d15bb40\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455"
Jul 28 10:25:39.940361 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:39.940343    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a58f731f-e46a-4cc4-b088-7dc00d15bb40-proxy-tls\") pod \"isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455\" (UID: \"a58f731f-e46a-4cc4-b088-7dc00d15bb40\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455"
Jul 28 10:25:40.159590 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:40.159557    2583 generic.go:356] "Generic (PLEG): container finished" podID="86f2d777-6f0c-445d-b9f1-d6b43e8ef110" containerID="3c756cebb30a7c49dce4c109bbb238bfe625f3dc4ef59c3d3cd9cb066cf568e1" exitCode=2
Jul 28 10:25:40.159590 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:40.159580    2583 generic.go:356] "Generic (PLEG): container finished" podID="86f2d777-6f0c-445d-b9f1-d6b43e8ef110" containerID="957a41e275fd1096ba6e08b5257ac43a90484f87e5bee0b595b7407d2de27f03" exitCode=2
Jul 28 10:25:40.160135 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:40.159606    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-8e237-predictor-8c874fbcc-fnrsp"
Jul 28 10:25:40.160135 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:40.159630    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-8e237-predictor-8c874fbcc-fnrsp" event={"ID":"86f2d777-6f0c-445d-b9f1-d6b43e8ef110","Type":"ContainerDied","Data":"3c756cebb30a7c49dce4c109bbb238bfe625f3dc4ef59c3d3cd9cb066cf568e1"}
Jul 28 10:25:40.160135 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:40.159689    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-8e237-predictor-8c874fbcc-fnrsp" event={"ID":"86f2d777-6f0c-445d-b9f1-d6b43e8ef110","Type":"ContainerDied","Data":"957a41e275fd1096ba6e08b5257ac43a90484f87e5bee0b595b7407d2de27f03"}
Jul 28 10:25:40.160135 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:40.159713    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-8e237-predictor-8c874fbcc-fnrsp" event={"ID":"86f2d777-6f0c-445d-b9f1-d6b43e8ef110","Type":"ContainerDied","Data":"f5f9476f844621d96abf42e26e5dc0a887af9abfa3434f153f45803d2150dcde"}
Jul 28 10:25:40.160135 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:40.159737    2583 scope.go:117] "RemoveContainer" containerID="3c756cebb30a7c49dce4c109bbb238bfe625f3dc4ef59c3d3cd9cb066cf568e1"
Jul 28 10:25:40.162230 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:40.162208    2583 generic.go:356] "Generic (PLEG): container finished" podID="0d96f7a4-d01d-4770-9a32-878cd03cf875" containerID="6ad42a0ccaa46289c4ab56cc95c7c0d63c1c5b854d312d5287bf1adb2454fe66" exitCode=2
Jul 28 10:25:40.162356 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:40.162281    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht" event={"ID":"0d96f7a4-d01d-4770-9a32-878cd03cf875","Type":"ContainerDied","Data":"6ad42a0ccaa46289c4ab56cc95c7c0d63c1c5b854d312d5287bf1adb2454fe66"}
Jul 28 10:25:40.168846 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:40.168832    2583 scope.go:117] "RemoveContainer" containerID="957a41e275fd1096ba6e08b5257ac43a90484f87e5bee0b595b7407d2de27f03"
Jul 28 10:25:40.175986 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:40.175969    2583 scope.go:117] "RemoveContainer" containerID="3c756cebb30a7c49dce4c109bbb238bfe625f3dc4ef59c3d3cd9cb066cf568e1"
Jul 28 10:25:40.176233 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:25:40.176216    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c756cebb30a7c49dce4c109bbb238bfe625f3dc4ef59c3d3cd9cb066cf568e1\": container with ID starting with 3c756cebb30a7c49dce4c109bbb238bfe625f3dc4ef59c3d3cd9cb066cf568e1 not found: ID does not exist" containerID="3c756cebb30a7c49dce4c109bbb238bfe625f3dc4ef59c3d3cd9cb066cf568e1"
Jul 28 10:25:40.176276 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:40.176243    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c756cebb30a7c49dce4c109bbb238bfe625f3dc4ef59c3d3cd9cb066cf568e1"} err="failed to get container status \"3c756cebb30a7c49dce4c109bbb238bfe625f3dc4ef59c3d3cd9cb066cf568e1\": rpc error: code = NotFound desc = could not find container \"3c756cebb30a7c49dce4c109bbb238bfe625f3dc4ef59c3d3cd9cb066cf568e1\": container with ID starting with 3c756cebb30a7c49dce4c109bbb238bfe625f3dc4ef59c3d3cd9cb066cf568e1 not found: ID does not exist"
Jul 28 10:25:40.176276 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:40.176260    2583 scope.go:117] "RemoveContainer" containerID="957a41e275fd1096ba6e08b5257ac43a90484f87e5bee0b595b7407d2de27f03"
Jul 28 10:25:40.176513 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:25:40.176497    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"957a41e275fd1096ba6e08b5257ac43a90484f87e5bee0b595b7407d2de27f03\": container with ID starting with 957a41e275fd1096ba6e08b5257ac43a90484f87e5bee0b595b7407d2de27f03 not found: ID does not exist" containerID="957a41e275fd1096ba6e08b5257ac43a90484f87e5bee0b595b7407d2de27f03"
Jul 28 10:25:40.176562 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:40.176517    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"957a41e275fd1096ba6e08b5257ac43a90484f87e5bee0b595b7407d2de27f03"} err="failed to get container status \"957a41e275fd1096ba6e08b5257ac43a90484f87e5bee0b595b7407d2de27f03\": rpc error: code = NotFound desc = could not find container \"957a41e275fd1096ba6e08b5257ac43a90484f87e5bee0b595b7407d2de27f03\": container with ID starting with 957a41e275fd1096ba6e08b5257ac43a90484f87e5bee0b595b7407d2de27f03 not found: ID does not exist"
Jul 28 10:25:40.176562 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:40.176534    2583 scope.go:117] "RemoveContainer" containerID="3c756cebb30a7c49dce4c109bbb238bfe625f3dc4ef59c3d3cd9cb066cf568e1"
Jul 28 10:25:40.176747 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:40.176728    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c756cebb30a7c49dce4c109bbb238bfe625f3dc4ef59c3d3cd9cb066cf568e1"} err="failed to get container status \"3c756cebb30a7c49dce4c109bbb238bfe625f3dc4ef59c3d3cd9cb066cf568e1\": rpc error: code = NotFound desc = could not find container \"3c756cebb30a7c49dce4c109bbb238bfe625f3dc4ef59c3d3cd9cb066cf568e1\": container with ID starting with 3c756cebb30a7c49dce4c109bbb238bfe625f3dc4ef59c3d3cd9cb066cf568e1 not found: ID does not exist"
Jul 28 10:25:40.176793 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:40.176748    2583 scope.go:117] "RemoveContainer" containerID="957a41e275fd1096ba6e08b5257ac43a90484f87e5bee0b595b7407d2de27f03"
Jul 28 10:25:40.176950 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:40.176933    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"957a41e275fd1096ba6e08b5257ac43a90484f87e5bee0b595b7407d2de27f03"} err="failed to get container status \"957a41e275fd1096ba6e08b5257ac43a90484f87e5bee0b595b7407d2de27f03\": rpc error: code = NotFound desc = could not find container \"957a41e275fd1096ba6e08b5257ac43a90484f87e5bee0b595b7407d2de27f03\": container with ID starting with 957a41e275fd1096ba6e08b5257ac43a90484f87e5bee0b595b7407d2de27f03 not found: ID does not exist"
Jul 28 10:25:40.182938 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:40.182917    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-8e237-predictor-8c874fbcc-fnrsp"]
Jul 28 10:25:40.186366 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:40.186343    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-8e237-predictor-8c874fbcc-fnrsp"]
Jul 28 10:25:40.216031 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:40.216004    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455"
Jul 28 10:25:40.350870 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:40.350846    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455"]
Jul 28 10:25:40.352937 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:25:40.352909    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda58f731f_e46a_4cc4_b088_7dc00d15bb40.slice/crio-f0ee8a871bbb9762eeb32e4db3ba2f0eab534bd3cab080e0957e9b9e87ae5ddc WatchSource:0}: Error finding container f0ee8a871bbb9762eeb32e4db3ba2f0eab534bd3cab080e0957e9b9e87ae5ddc: Status 404 returned error can't find the container with id f0ee8a871bbb9762eeb32e4db3ba2f0eab534bd3cab080e0957e9b9e87ae5ddc
Jul 28 10:25:41.169353 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:41.169310    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455" event={"ID":"a58f731f-e46a-4cc4-b088-7dc00d15bb40","Type":"ContainerStarted","Data":"142f18707e021605c7d928f5f2bea9309b167cd1ad2dac4deba62fa8676459e7"}
Jul 28 10:25:41.169778 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:41.169385    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455" event={"ID":"a58f731f-e46a-4cc4-b088-7dc00d15bb40","Type":"ContainerStarted","Data":"f0ee8a871bbb9762eeb32e4db3ba2f0eab534bd3cab080e0957e9b9e87ae5ddc"}
Jul 28 10:25:41.439496 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:41.439425    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86f2d777-6f0c-445d-b9f1-d6b43e8ef110" path="/var/lib/kubelet/pods/86f2d777-6f0c-445d-b9f1-d6b43e8ef110/volumes"
Jul 28 10:25:41.794429 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:41.794348    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht" podUID="0d96f7a4-d01d-4770-9a32-878cd03cf875" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Jul 28 10:25:44.185902 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:44.185868    2583 generic.go:356] "Generic (PLEG): container finished" podID="0d96f7a4-d01d-4770-9a32-878cd03cf875" containerID="fc6ecdef06e73643c30149e22425c72c6031117d401bfe79389558b25a00084e" exitCode=0
Jul 28 10:25:44.186282 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:44.185942    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht" event={"ID":"0d96f7a4-d01d-4770-9a32-878cd03cf875","Type":"ContainerDied","Data":"fc6ecdef06e73643c30149e22425c72c6031117d401bfe79389558b25a00084e"}
Jul 28 10:25:45.191438 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:45.191402    2583 generic.go:356] "Generic (PLEG): container finished" podID="a58f731f-e46a-4cc4-b088-7dc00d15bb40" containerID="142f18707e021605c7d928f5f2bea9309b167cd1ad2dac4deba62fa8676459e7" exitCode=0
Jul 28 10:25:45.191799 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:45.191455    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455" event={"ID":"a58f731f-e46a-4cc4-b088-7dc00d15bb40","Type":"ContainerDied","Data":"142f18707e021605c7d928f5f2bea9309b167cd1ad2dac4deba62fa8676459e7"}
Jul 28 10:25:46.203984 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:46.203950    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455" event={"ID":"a58f731f-e46a-4cc4-b088-7dc00d15bb40","Type":"ContainerStarted","Data":"134a2e22d6104ee2d7a7e8e4014d0ac1ce3976e31b2a922fc20ce21784a6a4f1"}
Jul 28 10:25:46.204395 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:46.204030    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455" event={"ID":"a58f731f-e46a-4cc4-b088-7dc00d15bb40","Type":"ContainerStarted","Data":"4b327a24faeb29d7c4352e1f461f105b4070e11915f3b4ddcbf6b90c0c48bea5"}
Jul 28 10:25:46.204395 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:46.204217    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455"
Jul 28 10:25:46.223526 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:46.223481    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455" podStartSLOduration=3.389554839 podStartE2EDuration="7.223467925s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.833913086s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:25:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:25:46.222366474 +0000 UTC m=+1085.322873029" watchObservedRunningTime="2026-07-28 10:25:46.223467925 +0000 UTC m=+1085.323974478"
Jul 28 10:25:46.794238 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:46.794171    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht" podUID="0d96f7a4-d01d-4770-9a32-878cd03cf875" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Jul 28 10:25:46.798498 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:46.798468    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht" podUID="0d96f7a4-d01d-4770-9a32-878cd03cf875" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 10:25:46.798800 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:46.798778    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht" podUID="0d96f7a4-d01d-4770-9a32-878cd03cf875" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 10:25:47.211739 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:47.211712    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455"
Jul 28 10:25:47.212795 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:47.212767    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455" podUID="a58f731f-e46a-4cc4-b088-7dc00d15bb40" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 10:25:48.215776 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:48.215730    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455" podUID="a58f731f-e46a-4cc4-b088-7dc00d15bb40" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 10:25:51.794533 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:51.794488    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht" podUID="0d96f7a4-d01d-4770-9a32-878cd03cf875" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Jul 28 10:25:51.795008 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:51.794596    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht"
Jul 28 10:25:53.220196 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:53.220157    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455"
Jul 28 10:25:53.226249 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:53.226206    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455" podUID="a58f731f-e46a-4cc4-b088-7dc00d15bb40" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 10:25:56.793781 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:56.793739    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht" podUID="0d96f7a4-d01d-4770-9a32-878cd03cf875" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Jul 28 10:25:56.799072 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:56.799044    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht" podUID="0d96f7a4-d01d-4770-9a32-878cd03cf875" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 10:25:56.799448 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:25:56.799424    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht" podUID="0d96f7a4-d01d-4770-9a32-878cd03cf875" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 10:26:01.794590 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:26:01.794541    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht" podUID="0d96f7a4-d01d-4770-9a32-878cd03cf875" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Jul 28 10:26:03.220944 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:26:03.220894    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455" podUID="a58f731f-e46a-4cc4-b088-7dc00d15bb40" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 10:26:06.794647 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:26:06.794602    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht" podUID="0d96f7a4-d01d-4770-9a32-878cd03cf875" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Jul 28 10:26:06.798982 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:26:06.798945    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht" podUID="0d96f7a4-d01d-4770-9a32-878cd03cf875" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 10:26:06.799127 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:26:06.799107    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht"
Jul 28 10:26:06.799304 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:26:06.799279    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht" podUID="0d96f7a4-d01d-4770-9a32-878cd03cf875" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 10:26:06.799389 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:26:06.799370    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht"
Jul 28 10:26:09.309705 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:26:09.309675    2583 generic.go:356] "Generic (PLEG): container finished" podID="0d96f7a4-d01d-4770-9a32-878cd03cf875" containerID="b1650c28b434981ab211532d0b7fd81aa5116df026e8145378197ee0c1f9b632" exitCode=0
Jul 28 10:26:09.310089 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:26:09.309748    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht" event={"ID":"0d96f7a4-d01d-4770-9a32-878cd03cf875","Type":"ContainerDied","Data":"b1650c28b434981ab211532d0b7fd81aa5116df026e8145378197ee0c1f9b632"}
Jul 28 10:26:09.335394 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:26:09.335359    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht"
Jul 28 10:26:09.485572 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:26:09.485482    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d96f7a4-d01d-4770-9a32-878cd03cf875-kserve-provision-location\") pod \"0d96f7a4-d01d-4770-9a32-878cd03cf875\" (UID: \"0d96f7a4-d01d-4770-9a32-878cd03cf875\") "
Jul 28 10:26:09.485572 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:26:09.485535    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d96f7a4-d01d-4770-9a32-878cd03cf875-kube-rbac-proxy-sar-config\") pod \"0d96f7a4-d01d-4770-9a32-878cd03cf875\" (UID: \"0d96f7a4-d01d-4770-9a32-878cd03cf875\") "
Jul 28 10:26:09.485572 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:26:09.485570    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d96f7a4-d01d-4770-9a32-878cd03cf875-cabundle-cert\") pod \"0d96f7a4-d01d-4770-9a32-878cd03cf875\" (UID: \"0d96f7a4-d01d-4770-9a32-878cd03cf875\") "
Jul 28 10:26:09.485853 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:26:09.485613    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d96f7a4-d01d-4770-9a32-878cd03cf875-proxy-tls\") pod \"0d96f7a4-d01d-4770-9a32-878cd03cf875\" (UID: \"0d96f7a4-d01d-4770-9a32-878cd03cf875\") "
Jul 28 10:26:09.485853 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:26:09.485704    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tvwl2\" (UniqueName: \"kubernetes.io/projected/0d96f7a4-d01d-4770-9a32-878cd03cf875-kube-api-access-tvwl2\") pod \"0d96f7a4-d01d-4770-9a32-878cd03cf875\" (UID: \"0d96f7a4-d01d-4770-9a32-878cd03cf875\") "
Jul 28 10:26:09.485853 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:26:09.485844    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d96f7a4-d01d-4770-9a32-878cd03cf875-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0d96f7a4-d01d-4770-9a32-878cd03cf875" (UID: "0d96f7a4-d01d-4770-9a32-878cd03cf875"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:26:09.486024 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:26:09.485926    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d96f7a4-d01d-4770-9a32-878cd03cf875-kserve-provision-location\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:26:09.486024 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:26:09.485973    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d96f7a4-d01d-4770-9a32-878cd03cf875-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0d96f7a4-d01d-4770-9a32-878cd03cf875" (UID: "0d96f7a4-d01d-4770-9a32-878cd03cf875"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:26:09.486024 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:26:09.485982    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d96f7a4-d01d-4770-9a32-878cd03cf875-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0d96f7a4-d01d-4770-9a32-878cd03cf875" (UID: "0d96f7a4-d01d-4770-9a32-878cd03cf875"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:26:09.487763 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:26:09.487737    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d96f7a4-d01d-4770-9a32-878cd03cf875-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0d96f7a4-d01d-4770-9a32-878cd03cf875" (UID: "0d96f7a4-d01d-4770-9a32-878cd03cf875"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:26:09.487763 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:26:09.487739    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d96f7a4-d01d-4770-9a32-878cd03cf875-kube-api-access-tvwl2" (OuterVolumeSpecName: "kube-api-access-tvwl2") pod "0d96f7a4-d01d-4770-9a32-878cd03cf875" (UID: "0d96f7a4-d01d-4770-9a32-878cd03cf875"). InnerVolumeSpecName "kube-api-access-tvwl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:26:09.587238 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:26:09.587198    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvwl2\" (UniqueName: \"kubernetes.io/projected/0d96f7a4-d01d-4770-9a32-878cd03cf875-kube-api-access-tvwl2\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:26:09.587238 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:26:09.587232    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d96f7a4-d01d-4770-9a32-878cd03cf875-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:26:09.587238 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:26:09.587243    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d96f7a4-d01d-4770-9a32-878cd03cf875-cabundle-cert\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:26:09.587474 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:26:09.587253    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d96f7a4-d01d-4770-9a32-878cd03cf875-proxy-tls\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:26:10.317056 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:26:10.317030    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht"
Jul 28 10:26:10.317511 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:26:10.317023    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht" event={"ID":"0d96f7a4-d01d-4770-9a32-878cd03cf875","Type":"ContainerDied","Data":"0c9b220e8eb7c732ceef82daf03cc4b0e03463ff02e30ff4c01a1149049659eb"}
Jul 28 10:26:10.317511 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:26:10.317203    2583 scope.go:117] "RemoveContainer" containerID="b1650c28b434981ab211532d0b7fd81aa5116df026e8145378197ee0c1f9b632"
Jul 28 10:26:10.325986 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:26:10.325968    2583 scope.go:117] "RemoveContainer" containerID="6ad42a0ccaa46289c4ab56cc95c7c0d63c1c5b854d312d5287bf1adb2454fe66"
Jul 28 10:26:10.335210 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:26:10.335167    2583 scope.go:117] "RemoveContainer" containerID="fc6ecdef06e73643c30149e22425c72c6031117d401bfe79389558b25a00084e"
Jul 28 10:26:10.341279 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:26:10.341251    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht"]
Jul 28 10:26:10.346899 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:26:10.346357    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-8e237-predictor-557d6ccdb9-kjfht"]
Jul 28 10:26:10.346899 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:26:10.346889    2583 scope.go:117] "RemoveContainer" containerID="a2b42f8068fd96683b80c1e2f0d3cdb1f71b19dae7ac5cdae503130aadae1fbf"
Jul 28 10:26:11.438148 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:26:11.438113    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d96f7a4-d01d-4770-9a32-878cd03cf875" path="/var/lib/kubelet/pods/0d96f7a4-d01d-4770-9a32-878cd03cf875/volumes"
Jul 28 10:26:13.221668 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:26:13.221622    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455" podUID="a58f731f-e46a-4cc4-b088-7dc00d15bb40" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 10:26:23.221157 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:26:23.221116    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455" podUID="a58f731f-e46a-4cc4-b088-7dc00d15bb40" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 10:26:24.150646 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:26:24.150611    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:26:33.221534 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:26:33.221487    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455" podUID="a58f731f-e46a-4cc4-b088-7dc00d15bb40" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 10:26:43.220975 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:26:43.220930    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455" podUID="a58f731f-e46a-4cc4-b088-7dc00d15bb40" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 10:26:53.220787 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:26:53.220741    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455" podUID="a58f731f-e46a-4cc4-b088-7dc00d15bb40" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 10:27:03.221633 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:27:03.221549    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455" podUID="a58f731f-e46a-4cc4-b088-7dc00d15bb40" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 10:27:03.433312 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:27:03.433270    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455" podUID="a58f731f-e46a-4cc4-b088-7dc00d15bb40" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 10:27:13.434249 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:27:13.434194    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455" podUID="a58f731f-e46a-4cc4-b088-7dc00d15bb40" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 10:27:23.433918 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:27:23.433873    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455" podUID="a58f731f-e46a-4cc4-b088-7dc00d15bb40" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 10:27:33.433953 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:27:33.433901    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455" podUID="a58f731f-e46a-4cc4-b088-7dc00d15bb40" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 10:27:43.434128 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:27:43.434079    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455" podUID="a58f731f-e46a-4cc4-b088-7dc00d15bb40" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 10:27:53.433545 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:27:53.433495    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455" podUID="a58f731f-e46a-4cc4-b088-7dc00d15bb40" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 10:28:03.434151 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:03.434100    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455" podUID="a58f731f-e46a-4cc4-b088-7dc00d15bb40" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 10:28:04.433588 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:04.433543    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455" podUID="a58f731f-e46a-4cc4-b088-7dc00d15bb40" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 10:28:14.434505 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:14.434460    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455" podUID="a58f731f-e46a-4cc4-b088-7dc00d15bb40" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 10:28:24.435056 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:24.434924    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455"
Jul 28 10:28:29.471639 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:29.471600    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455"]
Jul 28 10:28:29.472163 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:29.471825    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455" podUID="a58f731f-e46a-4cc4-b088-7dc00d15bb40" containerName="kserve-container" containerID="cri-o://4b327a24faeb29d7c4352e1f461f105b4070e11915f3b4ddcbf6b90c0c48bea5" gracePeriod=30
Jul 28 10:28:29.472163 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:29.471870    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455" podUID="a58f731f-e46a-4cc4-b088-7dc00d15bb40" containerName="kube-rbac-proxy" containerID="cri-o://134a2e22d6104ee2d7a7e8e4014d0ac1ce3976e31b2a922fc20ce21784a6a4f1" gracePeriod=30
Jul 28 10:28:29.509884 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:29.509852    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:28:29.940127 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:29.940093    2583 generic.go:356] "Generic (PLEG): container finished" podID="a58f731f-e46a-4cc4-b088-7dc00d15bb40" containerID="134a2e22d6104ee2d7a7e8e4014d0ac1ce3976e31b2a922fc20ce21784a6a4f1" exitCode=2
Jul 28 10:28:29.940327 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:29.940161    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455" event={"ID":"a58f731f-e46a-4cc4-b088-7dc00d15bb40","Type":"ContainerDied","Data":"134a2e22d6104ee2d7a7e8e4014d0ac1ce3976e31b2a922fc20ce21784a6a4f1"}
Jul 28 10:28:33.216170 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:33.216064    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455" podUID="a58f731f-e46a-4cc4-b088-7dc00d15bb40" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Jul 28 10:28:34.433498 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:34.433449    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455" podUID="a58f731f-e46a-4cc4-b088-7dc00d15bb40" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 10:28:34.603597 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:34.603556    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-qg8bw"]
Jul 28 10:28:34.604078 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:34.604063    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d96f7a4-d01d-4770-9a32-878cd03cf875" containerName="kube-rbac-proxy"
Jul 28 10:28:34.604127 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:34.604081    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d96f7a4-d01d-4770-9a32-878cd03cf875" containerName="kube-rbac-proxy"
Jul 28 10:28:34.604127 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:34.604099    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86f2d777-6f0c-445d-b9f1-d6b43e8ef110" containerName="kube-rbac-proxy"
Jul 28 10:28:34.604127 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:34.604108    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="86f2d777-6f0c-445d-b9f1-d6b43e8ef110" containerName="kube-rbac-proxy"
Jul 28 10:28:34.604127 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:34.604120    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d96f7a4-d01d-4770-9a32-878cd03cf875" containerName="kserve-container"
Jul 28 10:28:34.604362 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:34.604128    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d96f7a4-d01d-4770-9a32-878cd03cf875" containerName="kserve-container"
Jul 28 10:28:34.604362 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:34.604139    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86f2d777-6f0c-445d-b9f1-d6b43e8ef110" containerName="kserve-container"
Jul 28 10:28:34.604362 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:34.604147    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="86f2d777-6f0c-445d-b9f1-d6b43e8ef110" containerName="kserve-container"
Jul 28 10:28:34.604362 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:34.604164    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d96f7a4-d01d-4770-9a32-878cd03cf875" containerName="agent"
Jul 28 10:28:34.604362 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:34.604171    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d96f7a4-d01d-4770-9a32-878cd03cf875" containerName="agent"
Jul 28 10:28:34.604362 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:34.604200    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d96f7a4-d01d-4770-9a32-878cd03cf875" containerName="storage-initializer"
Jul 28 10:28:34.604362 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:34.604209    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d96f7a4-d01d-4770-9a32-878cd03cf875" containerName="storage-initializer"
Jul 28 10:28:34.604362 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:34.604280    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d96f7a4-d01d-4770-9a32-878cd03cf875" containerName="kube-rbac-proxy"
Jul 28 10:28:34.604362 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:34.604293    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d96f7a4-d01d-4770-9a32-878cd03cf875" containerName="kserve-container"
Jul 28 10:28:34.604362 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:34.604303    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d96f7a4-d01d-4770-9a32-878cd03cf875" containerName="agent"
Jul 28 10:28:34.604362 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:34.604312    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="86f2d777-6f0c-445d-b9f1-d6b43e8ef110" containerName="kserve-container"
Jul 28 10:28:34.604362 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:34.604322    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="86f2d777-6f0c-445d-b9f1-d6b43e8ef110" containerName="kube-rbac-proxy"
Jul 28 10:28:34.614276 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:34.614238    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-2bz7c"]
Jul 28 10:28:34.614442 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:34.614362    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-qg8bw"
Jul 28 10:28:34.616446 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:34.616416    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Jul 28 10:28:34.618698 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:34.618654    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-2bz7c"
Jul 28 10:28:34.619872 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:34.618854    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Jul 28 10:28:34.619872 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:34.619081    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-qg8bw"]
Jul 28 10:28:34.620688 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:34.620663    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Jul 28 10:28:34.621490 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:34.621459    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-2bz7c"]
Jul 28 10:28:34.626667 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:34.626638    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:28:34.687258 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:34.687153    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/753d4e8f-33ed-4b81-9534-cf2d5c6aa84a-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-5957db6ddf-2bz7c\" (UID: \"753d4e8f-33ed-4b81-9534-cf2d5c6aa84a\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-2bz7c"
Jul 28 10:28:34.687258 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:34.687231    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/753d4e8f-33ed-4b81-9534-cf2d5c6aa84a-proxy-tls\") pod \"isvc-canary-create-v2-predictor-5957db6ddf-2bz7c\" (UID: \"753d4e8f-33ed-4b81-9534-cf2d5c6aa84a\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-2bz7c"
Jul 28 10:28:34.687258 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:34.687253    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/753d4e8f-33ed-4b81-9534-cf2d5c6aa84a-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-5957db6ddf-2bz7c\" (UID: \"753d4e8f-33ed-4b81-9534-cf2d5c6aa84a\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-2bz7c"
Jul 28 10:28:34.687514 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:34.687303    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba995822-3592-44ef-9487-1679381bc42f-proxy-tls\") pod \"isvc-canary-create-predictor-69fc457c9b-qg8bw\" (UID: \"ba995822-3592-44ef-9487-1679381bc42f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-qg8bw"
Jul 28 10:28:34.687514 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:34.687319    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ba995822-3592-44ef-9487-1679381bc42f-cabundle-cert\") pod \"isvc-canary-create-predictor-69fc457c9b-qg8bw\" (UID: \"ba995822-3592-44ef-9487-1679381bc42f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-qg8bw"
Jul 28 10:28:34.687514 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:34.687387    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr5n5\" (UniqueName: \"kubernetes.io/projected/753d4e8f-33ed-4b81-9534-cf2d5c6aa84a-kube-api-access-kr5n5\") pod \"isvc-canary-create-v2-predictor-5957db6ddf-2bz7c\" (UID: \"753d4e8f-33ed-4b81-9534-cf2d5c6aa84a\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-2bz7c"
Jul 28 10:28:34.687514 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:34.687438    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/753d4e8f-33ed-4b81-9534-cf2d5c6aa84a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-5957db6ddf-2bz7c\" (UID: \"753d4e8f-33ed-4b81-9534-cf2d5c6aa84a\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-2bz7c"
Jul 28 10:28:34.687514 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:34.687459    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg549\" (UniqueName: \"kubernetes.io/projected/ba995822-3592-44ef-9487-1679381bc42f-kube-api-access-pg549\") pod \"isvc-canary-create-predictor-69fc457c9b-qg8bw\" (UID: \"ba995822-3592-44ef-9487-1679381bc42f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-qg8bw"
Jul 28 10:28:34.687514 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:34.687487    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ba995822-3592-44ef-9487-1679381bc42f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-69fc457c9b-qg8bw\" (UID: \"ba995822-3592-44ef-9487-1679381bc42f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-qg8bw"
Jul 28 10:28:34.687705 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:34.687520    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba995822-3592-44ef-9487-1679381bc42f-kserve-provision-location\") pod \"isvc-canary-create-predictor-69fc457c9b-qg8bw\" (UID: \"ba995822-3592-44ef-9487-1679381bc42f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-qg8bw"
Jul 28 10:28:34.725530 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:34.725498    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:28:34.788337 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:34.788293    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/753d4e8f-33ed-4b81-9534-cf2d5c6aa84a-proxy-tls\") pod \"isvc-canary-create-v2-predictor-5957db6ddf-2bz7c\" (UID: \"753d4e8f-33ed-4b81-9534-cf2d5c6aa84a\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-2bz7c"
Jul 28 10:28:34.788337 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:34.788339    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/753d4e8f-33ed-4b81-9534-cf2d5c6aa84a-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-5957db6ddf-2bz7c\" (UID: \"753d4e8f-33ed-4b81-9534-cf2d5c6aa84a\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-2bz7c"
Jul 28 10:28:34.788617 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:34.788375    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba995822-3592-44ef-9487-1679381bc42f-proxy-tls\") pod \"isvc-canary-create-predictor-69fc457c9b-qg8bw\" (UID: \"ba995822-3592-44ef-9487-1679381bc42f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-qg8bw"
Jul 28 10:28:34.788617 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:34.788396    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ba995822-3592-44ef-9487-1679381bc42f-cabundle-cert\") pod \"isvc-canary-create-predictor-69fc457c9b-qg8bw\" (UID: \"ba995822-3592-44ef-9487-1679381bc42f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-qg8bw"
Jul 28 10:28:34.788617 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:34.788422    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr5n5\" (UniqueName: \"kubernetes.io/projected/753d4e8f-33ed-4b81-9534-cf2d5c6aa84a-kube-api-access-kr5n5\") pod \"isvc-canary-create-v2-predictor-5957db6ddf-2bz7c\" (UID: \"753d4e8f-33ed-4b81-9534-cf2d5c6aa84a\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-2bz7c"
Jul 28 10:28:34.788617 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:34.788453    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/753d4e8f-33ed-4b81-9534-cf2d5c6aa84a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-5957db6ddf-2bz7c\" (UID: \"753d4e8f-33ed-4b81-9534-cf2d5c6aa84a\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-2bz7c"
Jul 28 10:28:34.788617 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:34.788475    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg549\" (UniqueName: \"kubernetes.io/projected/ba995822-3592-44ef-9487-1679381bc42f-kube-api-access-pg549\") pod \"isvc-canary-create-predictor-69fc457c9b-qg8bw\" (UID: \"ba995822-3592-44ef-9487-1679381bc42f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-qg8bw"
Jul 28 10:28:34.788617 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:34.788509    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ba995822-3592-44ef-9487-1679381bc42f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-69fc457c9b-qg8bw\" (UID: \"ba995822-3592-44ef-9487-1679381bc42f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-qg8bw"
Jul 28 10:28:34.788617 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:34.788551    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba995822-3592-44ef-9487-1679381bc42f-kserve-provision-location\") pod \"isvc-canary-create-predictor-69fc457c9b-qg8bw\" (UID: \"ba995822-3592-44ef-9487-1679381bc42f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-qg8bw"
Jul 28 10:28:34.788617 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:34.788581    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/753d4e8f-33ed-4b81-9534-cf2d5c6aa84a-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-5957db6ddf-2bz7c\" (UID: \"753d4e8f-33ed-4b81-9534-cf2d5c6aa84a\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-2bz7c"
Jul 28 10:28:34.789129 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:28:34.788473    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-v2-predictor-serving-cert: secret "isvc-canary-create-v2-predictor-serving-cert" not found
Jul 28 10:28:34.789129 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:34.788859    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/753d4e8f-33ed-4b81-9534-cf2d5c6aa84a-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-5957db6ddf-2bz7c\" (UID: \"753d4e8f-33ed-4b81-9534-cf2d5c6aa84a\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-2bz7c"
Jul 28 10:28:34.789129 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:28:34.788950    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/753d4e8f-33ed-4b81-9534-cf2d5c6aa84a-proxy-tls podName:753d4e8f-33ed-4b81-9534-cf2d5c6aa84a nodeName:}" failed. No retries permitted until 2026-07-28 10:28:35.28892158 +0000 UTC m=+1254.389428123 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/753d4e8f-33ed-4b81-9534-cf2d5c6aa84a-proxy-tls") pod "isvc-canary-create-v2-predictor-5957db6ddf-2bz7c" (UID: "753d4e8f-33ed-4b81-9534-cf2d5c6aa84a") : secret "isvc-canary-create-v2-predictor-serving-cert" not found
Jul 28 10:28:34.789378 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:34.789295    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/753d4e8f-33ed-4b81-9534-cf2d5c6aa84a-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-5957db6ddf-2bz7c\" (UID: \"753d4e8f-33ed-4b81-9534-cf2d5c6aa84a\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-2bz7c"
Jul 28 10:28:34.789378 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:34.789295    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ba995822-3592-44ef-9487-1679381bc42f-cabundle-cert\") pod \"isvc-canary-create-predictor-69fc457c9b-qg8bw\" (UID: \"ba995822-3592-44ef-9487-1679381bc42f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-qg8bw"
Jul 28 10:28:34.789378 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:34.789325    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ba995822-3592-44ef-9487-1679381bc42f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-69fc457c9b-qg8bw\" (UID: \"ba995822-3592-44ef-9487-1679381bc42f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-qg8bw"
Jul 28 10:28:34.789552 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:34.789445    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/753d4e8f-33ed-4b81-9534-cf2d5c6aa84a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-5957db6ddf-2bz7c\" (UID: \"753d4e8f-33ed-4b81-9534-cf2d5c6aa84a\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-2bz7c"
Jul 28 10:28:34.789612 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:34.789548    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba995822-3592-44ef-9487-1679381bc42f-kserve-provision-location\") pod \"isvc-canary-create-predictor-69fc457c9b-qg8bw\" (UID: \"ba995822-3592-44ef-9487-1679381bc42f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-qg8bw"
Jul 28 10:28:34.791051 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:34.791030    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba995822-3592-44ef-9487-1679381bc42f-proxy-tls\") pod \"isvc-canary-create-predictor-69fc457c9b-qg8bw\" (UID: \"ba995822-3592-44ef-9487-1679381bc42f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-qg8bw"
Jul 28 10:28:34.796762 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:34.796744    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr5n5\" (UniqueName: \"kubernetes.io/projected/753d4e8f-33ed-4b81-9534-cf2d5c6aa84a-kube-api-access-kr5n5\") pod \"isvc-canary-create-v2-predictor-5957db6ddf-2bz7c\" (UID: \"753d4e8f-33ed-4b81-9534-cf2d5c6aa84a\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-2bz7c"
Jul 28 10:28:34.797000 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:34.796984    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg549\" (UniqueName: \"kubernetes.io/projected/ba995822-3592-44ef-9487-1679381bc42f-kube-api-access-pg549\") pod \"isvc-canary-create-predictor-69fc457c9b-qg8bw\" (UID: \"ba995822-3592-44ef-9487-1679381bc42f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-qg8bw"
Jul 28 10:28:34.826418 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:34.826384    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:28:34.925785 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:34.925747    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:28:34.929649 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:34.929624    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-qg8bw"
Jul 28 10:28:35.068206 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:35.068165    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-qg8bw"]
Jul 28 10:28:35.070453 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:28:35.070422    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba995822_3592_44ef_9487_1679381bc42f.slice/crio-eadecb9da2afde46bdc926ed7c39c29f9089e6bba8ed3049f8f68a4a5356a3cf WatchSource:0}: Error finding container eadecb9da2afde46bdc926ed7c39c29f9089e6bba8ed3049f8f68a4a5356a3cf: Status 404 returned error can't find the container with id eadecb9da2afde46bdc926ed7c39c29f9089e6bba8ed3049f8f68a4a5356a3cf
Jul 28 10:28:35.293267 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:35.293163    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/753d4e8f-33ed-4b81-9534-cf2d5c6aa84a-proxy-tls\") pod \"isvc-canary-create-v2-predictor-5957db6ddf-2bz7c\" (UID: \"753d4e8f-33ed-4b81-9534-cf2d5c6aa84a\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-2bz7c"
Jul 28 10:28:35.295602 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:35.295567    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/753d4e8f-33ed-4b81-9534-cf2d5c6aa84a-proxy-tls\") pod \"isvc-canary-create-v2-predictor-5957db6ddf-2bz7c\" (UID: \"753d4e8f-33ed-4b81-9534-cf2d5c6aa84a\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-2bz7c"
Jul 28 10:28:35.538441 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:35.538404    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-2bz7c"
Jul 28 10:28:35.676058 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:35.676033    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-2bz7c"]
Jul 28 10:28:35.677963 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:28:35.677935    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod753d4e8f_33ed_4b81_9534_cf2d5c6aa84a.slice/crio-8adafaf8946cbd95cea92ae6fcab4af864634b5ed7615fe1ca8e943ac60ffab4 WatchSource:0}: Error finding container 8adafaf8946cbd95cea92ae6fcab4af864634b5ed7615fe1ca8e943ac60ffab4: Status 404 returned error can't find the container with id 8adafaf8946cbd95cea92ae6fcab4af864634b5ed7615fe1ca8e943ac60ffab4
Jul 28 10:28:35.968502 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:35.968461    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-2bz7c" event={"ID":"753d4e8f-33ed-4b81-9534-cf2d5c6aa84a","Type":"ContainerStarted","Data":"c38e72038769e0ad089e55e632c7a2e1aed67fbde25a2481386d7ce6b3dcfd3e"}
Jul 28 10:28:35.968845 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:35.968543    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-2bz7c" event={"ID":"753d4e8f-33ed-4b81-9534-cf2d5c6aa84a","Type":"ContainerStarted","Data":"8adafaf8946cbd95cea92ae6fcab4af864634b5ed7615fe1ca8e943ac60ffab4"}
Jul 28 10:28:35.970049 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:35.970023    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-qg8bw" event={"ID":"ba995822-3592-44ef-9487-1679381bc42f","Type":"ContainerStarted","Data":"efdc0328deca12b807a8d03fcef3d54b162a316b872820aa2f66dd49c036820e"}
Jul 28 10:28:35.970207 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:35.970073    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-qg8bw" event={"ID":"ba995822-3592-44ef-9487-1679381bc42f","Type":"ContainerStarted","Data":"eadecb9da2afde46bdc926ed7c39c29f9089e6bba8ed3049f8f68a4a5356a3cf"}
Jul 28 10:28:38.216097 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:38.216048    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455" podUID="a58f731f-e46a-4cc4-b088-7dc00d15bb40" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Jul 28 10:28:39.989697 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:39.989661    2583 generic.go:356] "Generic (PLEG): container finished" podID="753d4e8f-33ed-4b81-9534-cf2d5c6aa84a" containerID="c38e72038769e0ad089e55e632c7a2e1aed67fbde25a2481386d7ce6b3dcfd3e" exitCode=0
Jul 28 10:28:39.990202 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:39.989738    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-2bz7c" event={"ID":"753d4e8f-33ed-4b81-9534-cf2d5c6aa84a","Type":"ContainerDied","Data":"c38e72038769e0ad089e55e632c7a2e1aed67fbde25a2481386d7ce6b3dcfd3e"}
Jul 28 10:28:39.991374 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:39.991352    2583 generic.go:356] "Generic (PLEG): container finished" podID="ba995822-3592-44ef-9487-1679381bc42f" containerID="efdc0328deca12b807a8d03fcef3d54b162a316b872820aa2f66dd49c036820e" exitCode=0
Jul 28 10:28:39.991490 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:39.991422    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-qg8bw" event={"ID":"ba995822-3592-44ef-9487-1679381bc42f","Type":"ContainerDied","Data":"efdc0328deca12b807a8d03fcef3d54b162a316b872820aa2f66dd49c036820e"}
Jul 28 10:28:40.422924 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:40.422897    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455"
Jul 28 10:28:40.540351 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:40.540259    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4nj9c\" (UniqueName: \"kubernetes.io/projected/a58f731f-e46a-4cc4-b088-7dc00d15bb40-kube-api-access-4nj9c\") pod \"a58f731f-e46a-4cc4-b088-7dc00d15bb40\" (UID: \"a58f731f-e46a-4cc4-b088-7dc00d15bb40\") "
Jul 28 10:28:40.540351 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:40.540313    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a58f731f-e46a-4cc4-b088-7dc00d15bb40-kube-rbac-proxy-sar-config\") pod \"a58f731f-e46a-4cc4-b088-7dc00d15bb40\" (UID: \"a58f731f-e46a-4cc4-b088-7dc00d15bb40\") "
Jul 28 10:28:40.540351 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:40.540337    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a58f731f-e46a-4cc4-b088-7dc00d15bb40-cabundle-cert\") pod \"a58f731f-e46a-4cc4-b088-7dc00d15bb40\" (UID: \"a58f731f-e46a-4cc4-b088-7dc00d15bb40\") "
Jul 28 10:28:40.540578 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:40.540456    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a58f731f-e46a-4cc4-b088-7dc00d15bb40-kserve-provision-location\") pod \"a58f731f-e46a-4cc4-b088-7dc00d15bb40\" (UID: \"a58f731f-e46a-4cc4-b088-7dc00d15bb40\") "
Jul 28 10:28:40.540578 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:40.540512    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a58f731f-e46a-4cc4-b088-7dc00d15bb40-proxy-tls\") pod \"a58f731f-e46a-4cc4-b088-7dc00d15bb40\" (UID: \"a58f731f-e46a-4cc4-b088-7dc00d15bb40\") "
Jul 28 10:28:40.540706 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:40.540686    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a58f731f-e46a-4cc4-b088-7dc00d15bb40-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a58f731f-e46a-4cc4-b088-7dc00d15bb40" (UID: "a58f731f-e46a-4cc4-b088-7dc00d15bb40"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:28:40.540766 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:40.540750    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a58f731f-e46a-4cc4-b088-7dc00d15bb40-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a58f731f-e46a-4cc4-b088-7dc00d15bb40" (UID: "a58f731f-e46a-4cc4-b088-7dc00d15bb40"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:28:40.540817 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:40.540785    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a58f731f-e46a-4cc4-b088-7dc00d15bb40-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a58f731f-e46a-4cc4-b088-7dc00d15bb40" (UID: "a58f731f-e46a-4cc4-b088-7dc00d15bb40"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:28:40.541048 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:40.540998    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a58f731f-e46a-4cc4-b088-7dc00d15bb40-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:28:40.541048 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:40.541023    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a58f731f-e46a-4cc4-b088-7dc00d15bb40-cabundle-cert\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:28:40.541048 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:40.541038    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a58f731f-e46a-4cc4-b088-7dc00d15bb40-kserve-provision-location\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:28:40.542937 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:40.542909    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a58f731f-e46a-4cc4-b088-7dc00d15bb40-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a58f731f-e46a-4cc4-b088-7dc00d15bb40" (UID: "a58f731f-e46a-4cc4-b088-7dc00d15bb40"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:28:40.542937 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:40.542912    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a58f731f-e46a-4cc4-b088-7dc00d15bb40-kube-api-access-4nj9c" (OuterVolumeSpecName: "kube-api-access-4nj9c") pod "a58f731f-e46a-4cc4-b088-7dc00d15bb40" (UID: "a58f731f-e46a-4cc4-b088-7dc00d15bb40"). InnerVolumeSpecName "kube-api-access-4nj9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:28:40.641526 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:40.641489    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nj9c\" (UniqueName: \"kubernetes.io/projected/a58f731f-e46a-4cc4-b088-7dc00d15bb40-kube-api-access-4nj9c\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:28:40.641526 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:40.641522    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a58f731f-e46a-4cc4-b088-7dc00d15bb40-proxy-tls\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:28:40.998786 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:40.998746    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-2bz7c" event={"ID":"753d4e8f-33ed-4b81-9534-cf2d5c6aa84a","Type":"ContainerStarted","Data":"5eb60738f6dc5fcfb13d90980ab51f88c2ea5dd5b82699ce6cc2e3622da4b77e"}
Jul 28 10:28:40.999271 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:40.998821    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-2bz7c" event={"ID":"753d4e8f-33ed-4b81-9534-cf2d5c6aa84a","Type":"ContainerStarted","Data":"d93512b4764c2d58542f35f643eb824c85ba62667ca1ac57ffa97f9a76ad45f9"}
Jul 28 10:28:40.999271 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:40.999028    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-2bz7c"
Jul 28 10:28:41.000890 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:41.000868    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-qg8bw" event={"ID":"ba995822-3592-44ef-9487-1679381bc42f","Type":"ContainerStarted","Data":"e782a246ed00b4e46599b2ecc790bab2d1952fa72b9769a362eb62275efed480"}
Jul 28 10:28:41.000996 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:41.000916    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-qg8bw" event={"ID":"ba995822-3592-44ef-9487-1679381bc42f","Type":"ContainerStarted","Data":"b2cac5041868e715555aba35127bdc524219539c83c71ce4e4085e5585ac7423"}
Jul 28 10:28:41.001240 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:41.001209    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-qg8bw"
Jul 28 10:28:41.001369 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:41.001266    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-qg8bw"
Jul 28 10:28:41.002933 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:41.002902    2583 generic.go:356] "Generic (PLEG): container finished" podID="a58f731f-e46a-4cc4-b088-7dc00d15bb40" containerID="4b327a24faeb29d7c4352e1f461f105b4070e11915f3b4ddcbf6b90c0c48bea5" exitCode=0
Jul 28 10:28:41.003026 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:41.002937    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-qg8bw" podUID="ba995822-3592-44ef-9487-1679381bc42f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 10:28:41.003026 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:41.002955    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455"
Jul 28 10:28:41.003100 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:41.002953    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455" event={"ID":"a58f731f-e46a-4cc4-b088-7dc00d15bb40","Type":"ContainerDied","Data":"4b327a24faeb29d7c4352e1f461f105b4070e11915f3b4ddcbf6b90c0c48bea5"}
Jul 28 10:28:41.003100 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:41.003079    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455" event={"ID":"a58f731f-e46a-4cc4-b088-7dc00d15bb40","Type":"ContainerDied","Data":"f0ee8a871bbb9762eeb32e4db3ba2f0eab534bd3cab080e0957e9b9e87ae5ddc"}
Jul 28 10:28:41.003167 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:41.003126    2583 scope.go:117] "RemoveContainer" containerID="134a2e22d6104ee2d7a7e8e4014d0ac1ce3976e31b2a922fc20ce21784a6a4f1"
Jul 28 10:28:41.011584 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:41.011566    2583 scope.go:117] "RemoveContainer" containerID="4b327a24faeb29d7c4352e1f461f105b4070e11915f3b4ddcbf6b90c0c48bea5"
Jul 28 10:28:41.018805 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:41.018750    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-2bz7c" podStartSLOduration=3.514375388 podStartE2EDuration="7.01873081s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.504355422s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:28:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:28:41.01768717 +0000 UTC m=+1260.118193744" watchObservedRunningTime="2026-07-28 10:28:41.01873081 +0000 UTC m=+1260.119237365"
Jul 28 10:28:41.020271 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:41.020246    2583 scope.go:117] "RemoveContainer" containerID="142f18707e021605c7d928f5f2bea9309b167cd1ad2dac4deba62fa8676459e7"
Jul 28 10:28:41.028867 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:41.028843    2583 scope.go:117] "RemoveContainer" containerID="134a2e22d6104ee2d7a7e8e4014d0ac1ce3976e31b2a922fc20ce21784a6a4f1"
Jul 28 10:28:41.029213 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:28:41.029192    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"134a2e22d6104ee2d7a7e8e4014d0ac1ce3976e31b2a922fc20ce21784a6a4f1\": container with ID starting with 134a2e22d6104ee2d7a7e8e4014d0ac1ce3976e31b2a922fc20ce21784a6a4f1 not found: ID does not exist" containerID="134a2e22d6104ee2d7a7e8e4014d0ac1ce3976e31b2a922fc20ce21784a6a4f1"
Jul 28 10:28:41.029303 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:41.029222    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"134a2e22d6104ee2d7a7e8e4014d0ac1ce3976e31b2a922fc20ce21784a6a4f1"} err="failed to get container status \"134a2e22d6104ee2d7a7e8e4014d0ac1ce3976e31b2a922fc20ce21784a6a4f1\": rpc error: code = NotFound desc = could not find container \"134a2e22d6104ee2d7a7e8e4014d0ac1ce3976e31b2a922fc20ce21784a6a4f1\": container with ID starting with 134a2e22d6104ee2d7a7e8e4014d0ac1ce3976e31b2a922fc20ce21784a6a4f1 not found: ID does not exist"
Jul 28 10:28:41.029303 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:41.029241    2583 scope.go:117] "RemoveContainer" containerID="4b327a24faeb29d7c4352e1f461f105b4070e11915f3b4ddcbf6b90c0c48bea5"
Jul 28 10:28:41.029537 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:28:41.029518    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b327a24faeb29d7c4352e1f461f105b4070e11915f3b4ddcbf6b90c0c48bea5\": container with ID starting with 4b327a24faeb29d7c4352e1f461f105b4070e11915f3b4ddcbf6b90c0c48bea5 not found: ID does not exist" containerID="4b327a24faeb29d7c4352e1f461f105b4070e11915f3b4ddcbf6b90c0c48bea5"
Jul 28 10:28:41.029607 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:41.029542    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b327a24faeb29d7c4352e1f461f105b4070e11915f3b4ddcbf6b90c0c48bea5"} err="failed to get container status \"4b327a24faeb29d7c4352e1f461f105b4070e11915f3b4ddcbf6b90c0c48bea5\": rpc error: code = NotFound desc = could not find container \"4b327a24faeb29d7c4352e1f461f105b4070e11915f3b4ddcbf6b90c0c48bea5\": container with ID starting with 4b327a24faeb29d7c4352e1f461f105b4070e11915f3b4ddcbf6b90c0c48bea5 not found: ID does not exist"
Jul 28 10:28:41.029607 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:41.029556    2583 scope.go:117] "RemoveContainer" containerID="142f18707e021605c7d928f5f2bea9309b167cd1ad2dac4deba62fa8676459e7"
Jul 28 10:28:41.029817 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:28:41.029785    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"142f18707e021605c7d928f5f2bea9309b167cd1ad2dac4deba62fa8676459e7\": container with ID starting with 142f18707e021605c7d928f5f2bea9309b167cd1ad2dac4deba62fa8676459e7 not found: ID does not exist" containerID="142f18707e021605c7d928f5f2bea9309b167cd1ad2dac4deba62fa8676459e7"
Jul 28 10:28:41.029891 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:41.029825    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"142f18707e021605c7d928f5f2bea9309b167cd1ad2dac4deba62fa8676459e7"} err="failed to get container status \"142f18707e021605c7d928f5f2bea9309b167cd1ad2dac4deba62fa8676459e7\": rpc error: code = NotFound desc = could not find container \"142f18707e021605c7d928f5f2bea9309b167cd1ad2dac4deba62fa8676459e7\": container with ID starting with 142f18707e021605c7d928f5f2bea9309b167cd1ad2dac4deba62fa8676459e7 not found: ID does not exist"
Jul 28 10:28:41.037296 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:41.037241    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-qg8bw" podStartSLOduration=3.137114406 podStartE2EDuration="7.037223431s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.900109025s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:28:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:28:41.037220255 +0000 UTC m=+1260.137726811" watchObservedRunningTime="2026-07-28 10:28:41.037223431 +0000 UTC m=+1260.137729987"
Jul 28 10:28:41.050740 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:41.050715    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455"]
Jul 28 10:28:41.058387 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:41.058363    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-d2f58-predictor-55c88b88c5-mf455"]
Jul 28 10:28:41.439388 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:41.439350    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a58f731f-e46a-4cc4-b088-7dc00d15bb40" path="/var/lib/kubelet/pods/a58f731f-e46a-4cc4-b088-7dc00d15bb40/volumes"
Jul 28 10:28:42.007990 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:42.007942    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-qg8bw" podUID="ba995822-3592-44ef-9487-1679381bc42f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 10:28:42.007990 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:42.007985    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-2bz7c"
Jul 28 10:28:42.008737 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:42.008714    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-2bz7c" podUID="753d4e8f-33ed-4b81-9534-cf2d5c6aa84a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 10:28:43.012240 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:43.012199    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-2bz7c" podUID="753d4e8f-33ed-4b81-9534-cf2d5c6aa84a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 10:28:47.011657 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:47.011627    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-qg8bw"
Jul 28 10:28:47.012214 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:47.012106    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-qg8bw" podUID="ba995822-3592-44ef-9487-1679381bc42f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 10:28:48.016748 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:48.016715    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-2bz7c"
Jul 28 10:28:48.017366 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:48.017338    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-2bz7c" podUID="753d4e8f-33ed-4b81-9534-cf2d5c6aa84a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 10:28:48.622927 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:48.622898    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:28:48.902019 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:48.901993    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:28:57.012677 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:57.012628    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-qg8bw" podUID="ba995822-3592-44ef-9487-1679381bc42f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 10:28:58.017548 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:28:58.017506    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-2bz7c" podUID="753d4e8f-33ed-4b81-9534-cf2d5c6aa84a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 10:29:07.012581 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:29:07.012539    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-qg8bw" podUID="ba995822-3592-44ef-9487-1679381bc42f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 10:29:08.017385 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:29:08.017346    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-2bz7c" podUID="753d4e8f-33ed-4b81-9534-cf2d5c6aa84a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 10:29:14.719272 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:29:14.719230    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:29:17.012980 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:29:17.012938    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-qg8bw" podUID="ba995822-3592-44ef-9487-1679381bc42f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 10:29:18.017652 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:29:18.017606    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-2bz7c" podUID="753d4e8f-33ed-4b81-9534-cf2d5c6aa84a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 10:29:27.012760 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:29:27.012714    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-qg8bw" podUID="ba995822-3592-44ef-9487-1679381bc42f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 10:29:28.017273 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:29:28.017231    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-2bz7c" podUID="753d4e8f-33ed-4b81-9534-cf2d5c6aa84a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 10:29:37.012158 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:29:37.012094    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-qg8bw" podUID="ba995822-3592-44ef-9487-1679381bc42f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 10:29:38.018127 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:29:38.018088    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-2bz7c" podUID="753d4e8f-33ed-4b81-9534-cf2d5c6aa84a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 10:29:47.012689 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:29:47.012649    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-qg8bw" podUID="ba995822-3592-44ef-9487-1679381bc42f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 10:29:48.017921 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:29:48.017880    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-2bz7c" podUID="753d4e8f-33ed-4b81-9534-cf2d5c6aa84a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 10:29:57.012351 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:29:57.012320    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-qg8bw"
Jul 28 10:29:58.018241 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:29:58.018210    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-2bz7c"
Jul 28 10:30:04.690933 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:04.690845    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-qg8bw"]
Jul 28 10:30:04.691397 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:04.691231    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-qg8bw" podUID="ba995822-3592-44ef-9487-1679381bc42f" containerName="kserve-container" containerID="cri-o://b2cac5041868e715555aba35127bdc524219539c83c71ce4e4085e5585ac7423" gracePeriod=30
Jul 28 10:30:04.691397 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:04.691251    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-qg8bw" podUID="ba995822-3592-44ef-9487-1679381bc42f" containerName="kube-rbac-proxy" containerID="cri-o://e782a246ed00b4e46599b2ecc790bab2d1952fa72b9769a362eb62275efed480" gracePeriod=30
Jul 28 10:30:04.695243 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:04.695210    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-2bz7c"]
Jul 28 10:30:04.695439 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:04.695407    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-2bz7c" podUID="753d4e8f-33ed-4b81-9534-cf2d5c6aa84a" containerName="kserve-container" containerID="cri-o://d93512b4764c2d58542f35f643eb824c85ba62667ca1ac57ffa97f9a76ad45f9" gracePeriod=30
Jul 28 10:30:04.695881 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:04.695500    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-2bz7c" podUID="753d4e8f-33ed-4b81-9534-cf2d5c6aa84a" containerName="kube-rbac-proxy" containerID="cri-o://5eb60738f6dc5fcfb13d90980ab51f88c2ea5dd5b82699ce6cc2e3622da4b77e" gracePeriod=30
Jul 28 10:30:05.130513 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:05.130423    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:30:05.399627 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:05.399594    2583 generic.go:356] "Generic (PLEG): container finished" podID="753d4e8f-33ed-4b81-9534-cf2d5c6aa84a" containerID="5eb60738f6dc5fcfb13d90980ab51f88c2ea5dd5b82699ce6cc2e3622da4b77e" exitCode=2
Jul 28 10:30:05.399816 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:05.399671    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-2bz7c" event={"ID":"753d4e8f-33ed-4b81-9534-cf2d5c6aa84a","Type":"ContainerDied","Data":"5eb60738f6dc5fcfb13d90980ab51f88c2ea5dd5b82699ce6cc2e3622da4b77e"}
Jul 28 10:30:05.401657 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:05.401636    2583 generic.go:356] "Generic (PLEG): container finished" podID="ba995822-3592-44ef-9487-1679381bc42f" containerID="e782a246ed00b4e46599b2ecc790bab2d1952fa72b9769a362eb62275efed480" exitCode=2
Jul 28 10:30:05.401765 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:05.401705    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-qg8bw" event={"ID":"ba995822-3592-44ef-9487-1679381bc42f","Type":"ContainerDied","Data":"e782a246ed00b4e46599b2ecc790bab2d1952fa72b9769a362eb62275efed480"}
Jul 28 10:30:07.008152 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:07.008110    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-qg8bw" podUID="ba995822-3592-44ef-9487-1679381bc42f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Jul 28 10:30:07.012374 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:07.012345    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-qg8bw" podUID="ba995822-3592-44ef-9487-1679381bc42f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 10:30:07.207612 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:07.207580    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:30:08.013107 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:08.013064    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-2bz7c" podUID="753d4e8f-33ed-4b81-9534-cf2d5c6aa84a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Jul 28 10:30:08.017384 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:08.017361    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-2bz7c" podUID="753d4e8f-33ed-4b81-9534-cf2d5c6aa84a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 10:30:09.664498 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:09.664476    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-2bz7c"
Jul 28 10:30:09.667346 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:09.667327    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-qg8bw"
Jul 28 10:30:09.709872 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:09.709812    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/753d4e8f-33ed-4b81-9534-cf2d5c6aa84a-kserve-provision-location\") pod \"753d4e8f-33ed-4b81-9534-cf2d5c6aa84a\" (UID: \"753d4e8f-33ed-4b81-9534-cf2d5c6aa84a\") "
Jul 28 10:30:09.709872 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:09.709846    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/753d4e8f-33ed-4b81-9534-cf2d5c6aa84a-proxy-tls\") pod \"753d4e8f-33ed-4b81-9534-cf2d5c6aa84a\" (UID: \"753d4e8f-33ed-4b81-9534-cf2d5c6aa84a\") "
Jul 28 10:30:09.710073 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:09.709883    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pg549\" (UniqueName: \"kubernetes.io/projected/ba995822-3592-44ef-9487-1679381bc42f-kube-api-access-pg549\") pod \"ba995822-3592-44ef-9487-1679381bc42f\" (UID: \"ba995822-3592-44ef-9487-1679381bc42f\") "
Jul 28 10:30:09.710073 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:09.709908    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/753d4e8f-33ed-4b81-9534-cf2d5c6aa84a-cabundle-cert\") pod \"753d4e8f-33ed-4b81-9534-cf2d5c6aa84a\" (UID: \"753d4e8f-33ed-4b81-9534-cf2d5c6aa84a\") "
Jul 28 10:30:09.710073 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:09.710006    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ba995822-3592-44ef-9487-1679381bc42f-cabundle-cert\") pod \"ba995822-3592-44ef-9487-1679381bc42f\" (UID: \"ba995822-3592-44ef-9487-1679381bc42f\") "
Jul 28 10:30:09.710073 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:09.710061    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba995822-3592-44ef-9487-1679381bc42f-proxy-tls\") pod \"ba995822-3592-44ef-9487-1679381bc42f\" (UID: \"ba995822-3592-44ef-9487-1679381bc42f\") "
Jul 28 10:30:09.710305 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:09.710085    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kr5n5\" (UniqueName: \"kubernetes.io/projected/753d4e8f-33ed-4b81-9534-cf2d5c6aa84a-kube-api-access-kr5n5\") pod \"753d4e8f-33ed-4b81-9534-cf2d5c6aa84a\" (UID: \"753d4e8f-33ed-4b81-9534-cf2d5c6aa84a\") "
Jul 28 10:30:09.710305 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:09.710124    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/753d4e8f-33ed-4b81-9534-cf2d5c6aa84a-kube-rbac-proxy-sar-config\") pod \"753d4e8f-33ed-4b81-9534-cf2d5c6aa84a\" (UID: \"753d4e8f-33ed-4b81-9534-cf2d5c6aa84a\") "
Jul 28 10:30:09.710422 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:09.710405    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/753d4e8f-33ed-4b81-9534-cf2d5c6aa84a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "753d4e8f-33ed-4b81-9534-cf2d5c6aa84a" (UID: "753d4e8f-33ed-4b81-9534-cf2d5c6aa84a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:30:09.710477 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:09.710443    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/753d4e8f-33ed-4b81-9534-cf2d5c6aa84a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "753d4e8f-33ed-4b81-9534-cf2d5c6aa84a" (UID: "753d4e8f-33ed-4b81-9534-cf2d5c6aa84a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:30:09.710528 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:09.710490    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/753d4e8f-33ed-4b81-9534-cf2d5c6aa84a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "753d4e8f-33ed-4b81-9534-cf2d5c6aa84a" (UID: "753d4e8f-33ed-4b81-9534-cf2d5c6aa84a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:30:09.710528 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:09.710510    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba995822-3592-44ef-9487-1679381bc42f-kserve-provision-location\") pod \"ba995822-3592-44ef-9487-1679381bc42f\" (UID: \"ba995822-3592-44ef-9487-1679381bc42f\") "
Jul 28 10:30:09.710528 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:09.710512    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba995822-3592-44ef-9487-1679381bc42f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ba995822-3592-44ef-9487-1679381bc42f" (UID: "ba995822-3592-44ef-9487-1679381bc42f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:30:09.710693 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:09.710541    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ba995822-3592-44ef-9487-1679381bc42f-kube-rbac-proxy-sar-config\") pod \"ba995822-3592-44ef-9487-1679381bc42f\" (UID: \"ba995822-3592-44ef-9487-1679381bc42f\") "
Jul 28 10:30:09.710843 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:09.710825    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/753d4e8f-33ed-4b81-9534-cf2d5c6aa84a-kserve-provision-location\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:30:09.710918 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:09.710849    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/753d4e8f-33ed-4b81-9534-cf2d5c6aa84a-cabundle-cert\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:30:09.710918 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:09.710864    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ba995822-3592-44ef-9487-1679381bc42f-cabundle-cert\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:30:09.710918 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:09.710877    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/753d4e8f-33ed-4b81-9534-cf2d5c6aa84a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:30:09.711083 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:09.710999    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba995822-3592-44ef-9487-1679381bc42f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ba995822-3592-44ef-9487-1679381bc42f" (UID: "ba995822-3592-44ef-9487-1679381bc42f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:30:09.711083 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:09.711012    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba995822-3592-44ef-9487-1679381bc42f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ba995822-3592-44ef-9487-1679381bc42f" (UID: "ba995822-3592-44ef-9487-1679381bc42f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:30:09.712281 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:09.712257    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba995822-3592-44ef-9487-1679381bc42f-kube-api-access-pg549" (OuterVolumeSpecName: "kube-api-access-pg549") pod "ba995822-3592-44ef-9487-1679381bc42f" (UID: "ba995822-3592-44ef-9487-1679381bc42f"). InnerVolumeSpecName "kube-api-access-pg549". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:30:09.712370 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:09.712354    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/753d4e8f-33ed-4b81-9534-cf2d5c6aa84a-kube-api-access-kr5n5" (OuterVolumeSpecName: "kube-api-access-kr5n5") pod "753d4e8f-33ed-4b81-9534-cf2d5c6aa84a" (UID: "753d4e8f-33ed-4b81-9534-cf2d5c6aa84a"). InnerVolumeSpecName "kube-api-access-kr5n5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:30:09.712469 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:09.712449    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/753d4e8f-33ed-4b81-9534-cf2d5c6aa84a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "753d4e8f-33ed-4b81-9534-cf2d5c6aa84a" (UID: "753d4e8f-33ed-4b81-9534-cf2d5c6aa84a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:30:09.712721 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:09.712698    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba995822-3592-44ef-9487-1679381bc42f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ba995822-3592-44ef-9487-1679381bc42f" (UID: "ba995822-3592-44ef-9487-1679381bc42f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:30:09.779544 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:09.779513    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:30:09.811826 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:09.811796    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba995822-3592-44ef-9487-1679381bc42f-kserve-provision-location\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:30:09.811826 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:09.811823    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ba995822-3592-44ef-9487-1679381bc42f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:30:09.812066 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:09.811835    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/753d4e8f-33ed-4b81-9534-cf2d5c6aa84a-proxy-tls\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:30:09.812066 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:09.811847    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pg549\" (UniqueName: \"kubernetes.io/projected/ba995822-3592-44ef-9487-1679381bc42f-kube-api-access-pg549\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:30:09.812066 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:09.811855    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba995822-3592-44ef-9487-1679381bc42f-proxy-tls\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:30:09.812066 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:09.811864    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kr5n5\" (UniqueName: \"kubernetes.io/projected/753d4e8f-33ed-4b81-9534-cf2d5c6aa84a-kube-api-access-kr5n5\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:30:09.849389 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:09.849355    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-nctjt"]
Jul 28 10:30:09.849689 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:09.849677    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="753d4e8f-33ed-4b81-9534-cf2d5c6aa84a" containerName="kserve-container"
Jul 28 10:30:09.849734 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:09.849691    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="753d4e8f-33ed-4b81-9534-cf2d5c6aa84a" containerName="kserve-container"
Jul 28 10:30:09.849734 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:09.849700    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="753d4e8f-33ed-4b81-9534-cf2d5c6aa84a" containerName="storage-initializer"
Jul 28 10:30:09.849734 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:09.849707    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="753d4e8f-33ed-4b81-9534-cf2d5c6aa84a" containerName="storage-initializer"
Jul 28 10:30:09.849734 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:09.849718    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba995822-3592-44ef-9487-1679381bc42f" containerName="kube-rbac-proxy"
Jul 28 10:30:09.849734 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:09.849723    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba995822-3592-44ef-9487-1679381bc42f" containerName="kube-rbac-proxy"
Jul 28 10:30:09.849734 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:09.849733    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba995822-3592-44ef-9487-1679381bc42f" containerName="storage-initializer"
Jul 28 10:30:09.849918 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:09.849739    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba995822-3592-44ef-9487-1679381bc42f" containerName="storage-initializer"
Jul 28 10:30:09.849918 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:09.849748    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a58f731f-e46a-4cc4-b088-7dc00d15bb40" containerName="storage-initializer"
Jul 28 10:30:09.849918 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:09.849754    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a58f731f-e46a-4cc4-b088-7dc00d15bb40" containerName="storage-initializer"
Jul 28 10:30:09.849918 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:09.849762    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a58f731f-e46a-4cc4-b088-7dc00d15bb40" containerName="kube-rbac-proxy"
Jul 28 10:30:09.849918 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:09.849768    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a58f731f-e46a-4cc4-b088-7dc00d15bb40" containerName="kube-rbac-proxy"
Jul 28 10:30:09.849918 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:09.849775    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="753d4e8f-33ed-4b81-9534-cf2d5c6aa84a" containerName="kube-rbac-proxy"
Jul 28 10:30:09.849918 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:09.849780    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="753d4e8f-33ed-4b81-9534-cf2d5c6aa84a" containerName="kube-rbac-proxy"
Jul 28 10:30:09.849918 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:09.849790    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba995822-3592-44ef-9487-1679381bc42f" containerName="kserve-container"
Jul 28 10:30:09.849918 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:09.849795    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba995822-3592-44ef-9487-1679381bc42f" containerName="kserve-container"
Jul 28 10:30:09.849918 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:09.849801    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a58f731f-e46a-4cc4-b088-7dc00d15bb40" containerName="kserve-container"
Jul 28 10:30:09.849918 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:09.849806    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a58f731f-e46a-4cc4-b088-7dc00d15bb40" containerName="kserve-container"
Jul 28 10:30:09.849918 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:09.849872    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="753d4e8f-33ed-4b81-9534-cf2d5c6aa84a" containerName="kube-rbac-proxy"
Jul 28 10:30:09.849918 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:09.849882    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a58f731f-e46a-4cc4-b088-7dc00d15bb40" containerName="kube-rbac-proxy"
Jul 28 10:30:09.849918 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:09.849889    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba995822-3592-44ef-9487-1679381bc42f" containerName="kube-rbac-proxy"
Jul 28 10:30:09.849918 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:09.849896    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="753d4e8f-33ed-4b81-9534-cf2d5c6aa84a" containerName="kserve-container"
Jul 28 10:30:09.849918 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:09.849901    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba995822-3592-44ef-9487-1679381bc42f" containerName="kserve-container"
Jul 28 10:30:09.849918 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:09.849906    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a58f731f-e46a-4cc4-b088-7dc00d15bb40" containerName="kserve-container"
Jul 28 10:30:09.856204 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:09.854548    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:30:09.860503 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:09.860474    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r"]
Jul 28 10:30:09.861360 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:09.861336    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-nctjt"
Jul 28 10:30:09.863719 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:09.863694    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Jul 28 10:30:09.863891 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:09.863871    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Jul 28 10:30:09.865861 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:09.865564    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r"
Jul 28 10:30:09.867584 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:09.867561    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-nctjt"]
Jul 28 10:30:09.868109 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:09.868092    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Jul 28 10:30:09.871742 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:09.871331    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r"]
Jul 28 10:30:09.912686 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:09.912647    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8db4110e-b735-4859-8364-ab34da572d08-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r\" (UID: \"8db4110e-b735-4859-8364-ab34da572d08\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r"
Jul 28 10:30:09.912686 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:09.912689    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aff757f2-ce41-40bd-9510-763109a89559-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-698bbb5b56-nctjt\" (UID: \"aff757f2-ce41-40bd-9510-763109a89559\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-nctjt"
Jul 28 10:30:09.912952 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:09.912724    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8db4110e-b735-4859-8364-ab34da572d08-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r\" (UID: \"8db4110e-b735-4859-8364-ab34da572d08\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r"
Jul 28 10:30:09.912952 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:09.912753    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8db4110e-b735-4859-8364-ab34da572d08-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r\" (UID: \"8db4110e-b735-4859-8364-ab34da572d08\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r"
Jul 28 10:30:09.912952 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:09.912774    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aff757f2-ce41-40bd-9510-763109a89559-cabundle-cert\") pod \"isvc-canary-promote-predictor-698bbb5b56-nctjt\" (UID: \"aff757f2-ce41-40bd-9510-763109a89559\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-nctjt"
Jul 28 10:30:09.912952 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:09.912795    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8db4110e-b735-4859-8364-ab34da572d08-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r\" (UID: \"8db4110e-b735-4859-8364-ab34da572d08\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r"
Jul 28 10:30:09.912952 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:09.912864    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aff757f2-ce41-40bd-9510-763109a89559-proxy-tls\") pod \"isvc-canary-promote-predictor-698bbb5b56-nctjt\" (UID: \"aff757f2-ce41-40bd-9510-763109a89559\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-nctjt"
Jul 28 10:30:09.912952 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:09.912908    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aff757f2-ce41-40bd-9510-763109a89559-kserve-provision-location\") pod \"isvc-canary-promote-predictor-698bbb5b56-nctjt\" (UID: \"aff757f2-ce41-40bd-9510-763109a89559\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-nctjt"
Jul 28 10:30:09.913253 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:09.912961    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7vq6\" (UniqueName: \"kubernetes.io/projected/aff757f2-ce41-40bd-9510-763109a89559-kube-api-access-d7vq6\") pod \"isvc-canary-promote-predictor-698bbb5b56-nctjt\" (UID: \"aff757f2-ce41-40bd-9510-763109a89559\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-nctjt"
Jul 28 10:30:09.913253 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:09.912996    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8ckg\" (UniqueName: \"kubernetes.io/projected/8db4110e-b735-4859-8364-ab34da572d08-kube-api-access-j8ckg\") pod \"isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r\" (UID: \"8db4110e-b735-4859-8364-ab34da572d08\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r"
Jul 28 10:30:10.013646 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:10.013559    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8db4110e-b735-4859-8364-ab34da572d08-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r\" (UID: \"8db4110e-b735-4859-8364-ab34da572d08\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r"
Jul 28 10:30:10.013646 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:10.013598    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aff757f2-ce41-40bd-9510-763109a89559-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-698bbb5b56-nctjt\" (UID: \"aff757f2-ce41-40bd-9510-763109a89559\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-nctjt"
Jul 28 10:30:10.013646 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:10.013622    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8db4110e-b735-4859-8364-ab34da572d08-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r\" (UID: \"8db4110e-b735-4859-8364-ab34da572d08\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r"
Jul 28 10:30:10.014232 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:10.013674    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8db4110e-b735-4859-8364-ab34da572d08-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r\" (UID: \"8db4110e-b735-4859-8364-ab34da572d08\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r"
Jul 28 10:30:10.014232 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:10.013694    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aff757f2-ce41-40bd-9510-763109a89559-cabundle-cert\") pod \"isvc-canary-promote-predictor-698bbb5b56-nctjt\" (UID: \"aff757f2-ce41-40bd-9510-763109a89559\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-nctjt"
Jul 28 10:30:10.014232 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:10.013713    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8db4110e-b735-4859-8364-ab34da572d08-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r\" (UID: \"8db4110e-b735-4859-8364-ab34da572d08\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r"
Jul 28 10:30:10.014232 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:10.013757    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aff757f2-ce41-40bd-9510-763109a89559-proxy-tls\") pod \"isvc-canary-promote-predictor-698bbb5b56-nctjt\" (UID: \"aff757f2-ce41-40bd-9510-763109a89559\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-nctjt"
Jul 28 10:30:10.014232 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:10.013785    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aff757f2-ce41-40bd-9510-763109a89559-kserve-provision-location\") pod \"isvc-canary-promote-predictor-698bbb5b56-nctjt\" (UID: \"aff757f2-ce41-40bd-9510-763109a89559\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-nctjt"
Jul 28 10:30:10.014232 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:10.013817    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7vq6\" (UniqueName: \"kubernetes.io/projected/aff757f2-ce41-40bd-9510-763109a89559-kube-api-access-d7vq6\") pod \"isvc-canary-promote-predictor-698bbb5b56-nctjt\" (UID: \"aff757f2-ce41-40bd-9510-763109a89559\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-nctjt"
Jul 28 10:30:10.014232 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:10.013850    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8ckg\" (UniqueName: \"kubernetes.io/projected/8db4110e-b735-4859-8364-ab34da572d08-kube-api-access-j8ckg\") pod \"isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r\" (UID: \"8db4110e-b735-4859-8364-ab34da572d08\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r"
Jul 28 10:30:10.014232 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:30:10.013998    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-predictor-serving-cert: secret "isvc-canary-promote-predictor-serving-cert" not found
Jul 28 10:30:10.014232 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:30:10.014065    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aff757f2-ce41-40bd-9510-763109a89559-proxy-tls podName:aff757f2-ce41-40bd-9510-763109a89559 nodeName:}" failed. No retries permitted until 2026-07-28 10:30:10.514044379 +0000 UTC m=+1349.614550929 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/aff757f2-ce41-40bd-9510-763109a89559-proxy-tls") pod "isvc-canary-promote-predictor-698bbb5b56-nctjt" (UID: "aff757f2-ce41-40bd-9510-763109a89559") : secret "isvc-canary-promote-predictor-serving-cert" not found
Jul 28 10:30:10.014232 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:10.014123    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8db4110e-b735-4859-8364-ab34da572d08-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r\" (UID: \"8db4110e-b735-4859-8364-ab34da572d08\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r"
Jul 28 10:30:10.014783 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:10.014477    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8db4110e-b735-4859-8364-ab34da572d08-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r\" (UID: \"8db4110e-b735-4859-8364-ab34da572d08\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r"
Jul 28 10:30:10.014783 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:10.014477    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aff757f2-ce41-40bd-9510-763109a89559-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-698bbb5b56-nctjt\" (UID: \"aff757f2-ce41-40bd-9510-763109a89559\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-nctjt"
Jul 28 10:30:10.014783 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:10.014582    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aff757f2-ce41-40bd-9510-763109a89559-cabundle-cert\") pod \"isvc-canary-promote-predictor-698bbb5b56-nctjt\" (UID: \"aff757f2-ce41-40bd-9510-763109a89559\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-nctjt"
Jul 28 10:30:10.014783 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:10.014731    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aff757f2-ce41-40bd-9510-763109a89559-kserve-provision-location\") pod \"isvc-canary-promote-predictor-698bbb5b56-nctjt\" (UID: \"aff757f2-ce41-40bd-9510-763109a89559\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-nctjt"
Jul 28 10:30:10.014940 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:10.014874    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8db4110e-b735-4859-8364-ab34da572d08-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r\" (UID: \"8db4110e-b735-4859-8364-ab34da572d08\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r"
Jul 28 10:30:10.016759 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:10.016737    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8db4110e-b735-4859-8364-ab34da572d08-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r\" (UID: \"8db4110e-b735-4859-8364-ab34da572d08\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r"
Jul 28 10:30:10.024328 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:10.024309    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7vq6\" (UniqueName: \"kubernetes.io/projected/aff757f2-ce41-40bd-9510-763109a89559-kube-api-access-d7vq6\") pod \"isvc-canary-promote-predictor-698bbb5b56-nctjt\" (UID: \"aff757f2-ce41-40bd-9510-763109a89559\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-nctjt"
Jul 28 10:30:10.024541 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:10.024523    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8ckg\" (UniqueName: \"kubernetes.io/projected/8db4110e-b735-4859-8364-ab34da572d08-kube-api-access-j8ckg\") pod \"isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r\" (UID: \"8db4110e-b735-4859-8364-ab34da572d08\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r"
Jul 28 10:30:10.048696 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:10.048670    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:30:10.182069 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:10.182034    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r"
Jul 28 10:30:10.313773 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:10.313746    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r"]
Jul 28 10:30:10.315670 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:30:10.315645    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8db4110e_b735_4859_8364_ab34da572d08.slice/crio-557d8d0c6781fab835ec2150720f9428f39f98cb95cb948f0069d2b7507880c9 WatchSource:0}: Error finding container 557d8d0c6781fab835ec2150720f9428f39f98cb95cb948f0069d2b7507880c9: Status 404 returned error can't find the container with id 557d8d0c6781fab835ec2150720f9428f39f98cb95cb948f0069d2b7507880c9
Jul 28 10:30:10.368797 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:10.368774    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:30:10.425400 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:10.425366    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r" event={"ID":"8db4110e-b735-4859-8364-ab34da572d08","Type":"ContainerStarted","Data":"99111529909d5139873b5d5c5c3e0fc97a13d12edd6c7e3e3a44169a85a5d210"}
Jul 28 10:30:10.425746 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:10.425442    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r" event={"ID":"8db4110e-b735-4859-8364-ab34da572d08","Type":"ContainerStarted","Data":"557d8d0c6781fab835ec2150720f9428f39f98cb95cb948f0069d2b7507880c9"}
Jul 28 10:30:10.427420 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:10.427393    2583 generic.go:356] "Generic (PLEG): container finished" podID="753d4e8f-33ed-4b81-9534-cf2d5c6aa84a" containerID="d93512b4764c2d58542f35f643eb824c85ba62667ca1ac57ffa97f9a76ad45f9" exitCode=0
Jul 28 10:30:10.427551 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:10.427462    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-2bz7c" event={"ID":"753d4e8f-33ed-4b81-9534-cf2d5c6aa84a","Type":"ContainerDied","Data":"d93512b4764c2d58542f35f643eb824c85ba62667ca1ac57ffa97f9a76ad45f9"}
Jul 28 10:30:10.427551 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:10.427469    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-2bz7c"
Jul 28 10:30:10.427551 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:10.427512    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-2bz7c" event={"ID":"753d4e8f-33ed-4b81-9534-cf2d5c6aa84a","Type":"ContainerDied","Data":"8adafaf8946cbd95cea92ae6fcab4af864634b5ed7615fe1ca8e943ac60ffab4"}
Jul 28 10:30:10.427551 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:10.427548    2583 scope.go:117] "RemoveContainer" containerID="5eb60738f6dc5fcfb13d90980ab51f88c2ea5dd5b82699ce6cc2e3622da4b77e"
Jul 28 10:30:10.429783 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:10.429759    2583 generic.go:356] "Generic (PLEG): container finished" podID="ba995822-3592-44ef-9487-1679381bc42f" containerID="b2cac5041868e715555aba35127bdc524219539c83c71ce4e4085e5585ac7423" exitCode=0
Jul 28 10:30:10.429908 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:10.429799    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-qg8bw" event={"ID":"ba995822-3592-44ef-9487-1679381bc42f","Type":"ContainerDied","Data":"b2cac5041868e715555aba35127bdc524219539c83c71ce4e4085e5585ac7423"}
Jul 28 10:30:10.429908 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:10.429820    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-qg8bw"
Jul 28 10:30:10.429908 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:10.429851    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-qg8bw" event={"ID":"ba995822-3592-44ef-9487-1679381bc42f","Type":"ContainerDied","Data":"eadecb9da2afde46bdc926ed7c39c29f9089e6bba8ed3049f8f68a4a5356a3cf"}
Jul 28 10:30:10.450230 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:10.450207    2583 scope.go:117] "RemoveContainer" containerID="d93512b4764c2d58542f35f643eb824c85ba62667ca1ac57ffa97f9a76ad45f9"
Jul 28 10:30:10.459551 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:10.459532    2583 scope.go:117] "RemoveContainer" containerID="c38e72038769e0ad089e55e632c7a2e1aed67fbde25a2481386d7ce6b3dcfd3e"
Jul 28 10:30:10.464786 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:10.464760    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-2bz7c"]
Jul 28 10:30:10.467819 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:10.467799    2583 scope.go:117] "RemoveContainer" containerID="5eb60738f6dc5fcfb13d90980ab51f88c2ea5dd5b82699ce6cc2e3622da4b77e"
Jul 28 10:30:10.468109 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:30:10.468081    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5eb60738f6dc5fcfb13d90980ab51f88c2ea5dd5b82699ce6cc2e3622da4b77e\": container with ID starting with 5eb60738f6dc5fcfb13d90980ab51f88c2ea5dd5b82699ce6cc2e3622da4b77e not found: ID does not exist" containerID="5eb60738f6dc5fcfb13d90980ab51f88c2ea5dd5b82699ce6cc2e3622da4b77e"
Jul 28 10:30:10.468212 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:10.468120    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5eb60738f6dc5fcfb13d90980ab51f88c2ea5dd5b82699ce6cc2e3622da4b77e"} err="failed to get container status \"5eb60738f6dc5fcfb13d90980ab51f88c2ea5dd5b82699ce6cc2e3622da4b77e\": rpc error: code = NotFound desc = could not find container \"5eb60738f6dc5fcfb13d90980ab51f88c2ea5dd5b82699ce6cc2e3622da4b77e\": container with ID starting with 5eb60738f6dc5fcfb13d90980ab51f88c2ea5dd5b82699ce6cc2e3622da4b77e not found: ID does not exist"
Jul 28 10:30:10.468212 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:10.468139    2583 scope.go:117] "RemoveContainer" containerID="d93512b4764c2d58542f35f643eb824c85ba62667ca1ac57ffa97f9a76ad45f9"
Jul 28 10:30:10.468422 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:30:10.468407    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d93512b4764c2d58542f35f643eb824c85ba62667ca1ac57ffa97f9a76ad45f9\": container with ID starting with d93512b4764c2d58542f35f643eb824c85ba62667ca1ac57ffa97f9a76ad45f9 not found: ID does not exist" containerID="d93512b4764c2d58542f35f643eb824c85ba62667ca1ac57ffa97f9a76ad45f9"
Jul 28 10:30:10.468464 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:10.468427    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d93512b4764c2d58542f35f643eb824c85ba62667ca1ac57ffa97f9a76ad45f9"} err="failed to get container status \"d93512b4764c2d58542f35f643eb824c85ba62667ca1ac57ffa97f9a76ad45f9\": rpc error: code = NotFound desc = could not find container \"d93512b4764c2d58542f35f643eb824c85ba62667ca1ac57ffa97f9a76ad45f9\": container with ID starting with d93512b4764c2d58542f35f643eb824c85ba62667ca1ac57ffa97f9a76ad45f9 not found: ID does not exist"
Jul 28 10:30:10.468464 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:10.468442    2583 scope.go:117] "RemoveContainer" containerID="c38e72038769e0ad089e55e632c7a2e1aed67fbde25a2481386d7ce6b3dcfd3e"
Jul 28 10:30:10.468715 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:30:10.468691    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c38e72038769e0ad089e55e632c7a2e1aed67fbde25a2481386d7ce6b3dcfd3e\": container with ID starting with c38e72038769e0ad089e55e632c7a2e1aed67fbde25a2481386d7ce6b3dcfd3e not found: ID does not exist" containerID="c38e72038769e0ad089e55e632c7a2e1aed67fbde25a2481386d7ce6b3dcfd3e"
Jul 28 10:30:10.468766 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:10.468726    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c38e72038769e0ad089e55e632c7a2e1aed67fbde25a2481386d7ce6b3dcfd3e"} err="failed to get container status \"c38e72038769e0ad089e55e632c7a2e1aed67fbde25a2481386d7ce6b3dcfd3e\": rpc error: code = NotFound desc = could not find container \"c38e72038769e0ad089e55e632c7a2e1aed67fbde25a2481386d7ce6b3dcfd3e\": container with ID starting with c38e72038769e0ad089e55e632c7a2e1aed67fbde25a2481386d7ce6b3dcfd3e not found: ID does not exist"
Jul 28 10:30:10.468766 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:10.468752    2583 scope.go:117] "RemoveContainer" containerID="e782a246ed00b4e46599b2ecc790bab2d1952fa72b9769a362eb62275efed480"
Jul 28 10:30:10.468848 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:10.468818    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-2bz7c"]
Jul 28 10:30:10.477749 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:10.477733    2583 scope.go:117] "RemoveContainer" containerID="b2cac5041868e715555aba35127bdc524219539c83c71ce4e4085e5585ac7423"
Jul 28 10:30:10.479034 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:10.479014    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-qg8bw"]
Jul 28 10:30:10.483561 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:10.483540    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-qg8bw"]
Jul 28 10:30:10.485749 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:10.485725    2583 scope.go:117] "RemoveContainer" containerID="efdc0328deca12b807a8d03fcef3d54b162a316b872820aa2f66dd49c036820e"
Jul 28 10:30:10.494401 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:10.494381    2583 scope.go:117] "RemoveContainer" containerID="e782a246ed00b4e46599b2ecc790bab2d1952fa72b9769a362eb62275efed480"
Jul 28 10:30:10.494652 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:30:10.494632    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e782a246ed00b4e46599b2ecc790bab2d1952fa72b9769a362eb62275efed480\": container with ID starting with e782a246ed00b4e46599b2ecc790bab2d1952fa72b9769a362eb62275efed480 not found: ID does not exist" containerID="e782a246ed00b4e46599b2ecc790bab2d1952fa72b9769a362eb62275efed480"
Jul 28 10:30:10.494717 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:10.494657    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e782a246ed00b4e46599b2ecc790bab2d1952fa72b9769a362eb62275efed480"} err="failed to get container status \"e782a246ed00b4e46599b2ecc790bab2d1952fa72b9769a362eb62275efed480\": rpc error: code = NotFound desc = could not find container \"e782a246ed00b4e46599b2ecc790bab2d1952fa72b9769a362eb62275efed480\": container with ID starting with e782a246ed00b4e46599b2ecc790bab2d1952fa72b9769a362eb62275efed480 not found: ID does not exist"
Jul 28 10:30:10.494717 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:10.494676    2583 scope.go:117] "RemoveContainer" containerID="b2cac5041868e715555aba35127bdc524219539c83c71ce4e4085e5585ac7423"
Jul 28 10:30:10.494996 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:30:10.494978    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2cac5041868e715555aba35127bdc524219539c83c71ce4e4085e5585ac7423\": container with ID starting with b2cac5041868e715555aba35127bdc524219539c83c71ce4e4085e5585ac7423 not found: ID does not exist" containerID="b2cac5041868e715555aba35127bdc524219539c83c71ce4e4085e5585ac7423"
Jul 28 10:30:10.495042 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:10.495008    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2cac5041868e715555aba35127bdc524219539c83c71ce4e4085e5585ac7423"} err="failed to get container status \"b2cac5041868e715555aba35127bdc524219539c83c71ce4e4085e5585ac7423\": rpc error: code = NotFound desc = could not find container \"b2cac5041868e715555aba35127bdc524219539c83c71ce4e4085e5585ac7423\": container with ID starting with b2cac5041868e715555aba35127bdc524219539c83c71ce4e4085e5585ac7423 not found: ID does not exist"
Jul 28 10:30:10.495042 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:10.495024    2583 scope.go:117] "RemoveContainer" containerID="efdc0328deca12b807a8d03fcef3d54b162a316b872820aa2f66dd49c036820e"
Jul 28 10:30:10.495281 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:30:10.495266    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efdc0328deca12b807a8d03fcef3d54b162a316b872820aa2f66dd49c036820e\": container with ID starting with efdc0328deca12b807a8d03fcef3d54b162a316b872820aa2f66dd49c036820e not found: ID does not exist" containerID="efdc0328deca12b807a8d03fcef3d54b162a316b872820aa2f66dd49c036820e"
Jul 28 10:30:10.495343 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:10.495287    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efdc0328deca12b807a8d03fcef3d54b162a316b872820aa2f66dd49c036820e"} err="failed to get container status \"efdc0328deca12b807a8d03fcef3d54b162a316b872820aa2f66dd49c036820e\": rpc error: code = NotFound desc = could not find container \"efdc0328deca12b807a8d03fcef3d54b162a316b872820aa2f66dd49c036820e\": container with ID starting with efdc0328deca12b807a8d03fcef3d54b162a316b872820aa2f66dd49c036820e not found: ID does not exist"
Jul 28 10:30:10.518878 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:10.518851    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aff757f2-ce41-40bd-9510-763109a89559-proxy-tls\") pod \"isvc-canary-promote-predictor-698bbb5b56-nctjt\" (UID: \"aff757f2-ce41-40bd-9510-763109a89559\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-nctjt"
Jul 28 10:30:10.521081 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:10.521057    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aff757f2-ce41-40bd-9510-763109a89559-proxy-tls\") pod \"isvc-canary-promote-predictor-698bbb5b56-nctjt\" (UID: \"aff757f2-ce41-40bd-9510-763109a89559\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-nctjt"
Jul 28 10:30:10.776751 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:10.776709    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-nctjt"
Jul 28 10:30:10.914554 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:10.914531    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-nctjt"]
Jul 28 10:30:10.917035 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:30:10.916993    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaff757f2_ce41_40bd_9510_763109a89559.slice/crio-98af2267bf60add5282e90947c5d193aff1c64c1cd9364187592c7e09ffb65c8 WatchSource:0}: Error finding container 98af2267bf60add5282e90947c5d193aff1c64c1cd9364187592c7e09ffb65c8: Status 404 returned error can't find the container with id 98af2267bf60add5282e90947c5d193aff1c64c1cd9364187592c7e09ffb65c8
Jul 28 10:30:11.439616 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:11.439578    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="753d4e8f-33ed-4b81-9534-cf2d5c6aa84a" path="/var/lib/kubelet/pods/753d4e8f-33ed-4b81-9534-cf2d5c6aa84a/volumes"
Jul 28 10:30:11.440231 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:11.440195    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba995822-3592-44ef-9487-1679381bc42f" path="/var/lib/kubelet/pods/ba995822-3592-44ef-9487-1679381bc42f/volumes"
Jul 28 10:30:11.440605 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:11.440577    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-nctjt" event={"ID":"aff757f2-ce41-40bd-9510-763109a89559","Type":"ContainerStarted","Data":"548ea479da5eaed794a0404e22f7a9b79c65e59a5136667666e48fdf64527919"}
Jul 28 10:30:11.440682 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:11.440625    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-nctjt" event={"ID":"aff757f2-ce41-40bd-9510-763109a89559","Type":"ContainerStarted","Data":"98af2267bf60add5282e90947c5d193aff1c64c1cd9364187592c7e09ffb65c8"}
Jul 28 10:30:14.456159 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:14.456122    2583 generic.go:356] "Generic (PLEG): container finished" podID="8db4110e-b735-4859-8364-ab34da572d08" containerID="99111529909d5139873b5d5c5c3e0fc97a13d12edd6c7e3e3a44169a85a5d210" exitCode=0
Jul 28 10:30:14.456562 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:14.456198    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r" event={"ID":"8db4110e-b735-4859-8364-ab34da572d08","Type":"ContainerDied","Data":"99111529909d5139873b5d5c5c3e0fc97a13d12edd6c7e3e3a44169a85a5d210"}
Jul 28 10:30:15.463216 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:15.463170    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r" event={"ID":"8db4110e-b735-4859-8364-ab34da572d08","Type":"ContainerStarted","Data":"ca8f84e5e8efa2cf691b47c0b31841ec642240f5def2e2f7f50db0be27c2a7c5"}
Jul 28 10:30:15.463627 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:15.463247    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r" event={"ID":"8db4110e-b735-4859-8364-ab34da572d08","Type":"ContainerStarted","Data":"bacd9265b7ee67025a079b68bd816dfad047c9e4dd7a65cabedc3f176ffa93bb"}
Jul 28 10:30:15.463627 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:15.463502    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r"
Jul 28 10:30:15.463756 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:15.463652    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r"
Jul 28 10:30:15.464767 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:15.464743    2583 generic.go:356] "Generic (PLEG): container finished" podID="aff757f2-ce41-40bd-9510-763109a89559" containerID="548ea479da5eaed794a0404e22f7a9b79c65e59a5136667666e48fdf64527919" exitCode=0
Jul 28 10:30:15.464875 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:15.464819    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-nctjt" event={"ID":"aff757f2-ce41-40bd-9510-763109a89559","Type":"ContainerDied","Data":"548ea479da5eaed794a0404e22f7a9b79c65e59a5136667666e48fdf64527919"}
Jul 28 10:30:15.465050 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:15.465027    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r" podUID="8db4110e-b735-4859-8364-ab34da572d08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 10:30:15.484552 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:15.484514    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r" podStartSLOduration=2.945288978 podStartE2EDuration="6.484499807s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.539210829s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:30:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:30:15.482948612 +0000 UTC m=+1354.583455193" watchObservedRunningTime="2026-07-28 10:30:15.484499807 +0000 UTC m=+1354.585006361"
Jul 28 10:30:16.472055 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:16.472013    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-nctjt" event={"ID":"aff757f2-ce41-40bd-9510-763109a89559","Type":"ContainerStarted","Data":"2c5b4348ede083ac4d3868a43e1f2f0b4ae34ab15861d35a9622dec78dc2311d"}
Jul 28 10:30:16.472467 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:16.472085    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-nctjt" event={"ID":"aff757f2-ce41-40bd-9510-763109a89559","Type":"ContainerStarted","Data":"f682a7f20c17f36f43d56ea3eccf9e2a5c3ebb90332ead76934fe6ac9abb1b2a"}
Jul 28 10:30:16.472529 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:16.472497    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r" podUID="8db4110e-b735-4859-8364-ab34da572d08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 10:30:16.472579 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:16.472553    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-nctjt"
Jul 28 10:30:16.472698 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:16.472683    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-nctjt"
Jul 28 10:30:16.473846 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:16.473819    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-nctjt" podUID="aff757f2-ce41-40bd-9510-763109a89559" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 10:30:16.491540 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:16.491497    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-nctjt" podStartSLOduration=3.9027811740000002 podStartE2EDuration="7.491483622s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.588702448s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:30:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:30:16.489363494 +0000 UTC m=+1355.589870048" watchObservedRunningTime="2026-07-28 10:30:16.491483622 +0000 UTC m=+1355.591990176"
Jul 28 10:30:17.476514 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:17.476474    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-nctjt" podUID="aff757f2-ce41-40bd-9510-763109a89559" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 10:30:21.477316 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:21.477284    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r"
Jul 28 10:30:21.477868 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:21.477838    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r" podUID="8db4110e-b735-4859-8364-ab34da572d08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 10:30:22.480471 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:22.480442    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-nctjt"
Jul 28 10:30:22.480936 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:22.480909    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-nctjt" podUID="aff757f2-ce41-40bd-9510-763109a89559" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 10:30:31.477771 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:31.477724    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r" podUID="8db4110e-b735-4859-8364-ab34da572d08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 10:30:32.481775 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:32.481734    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-nctjt" podUID="aff757f2-ce41-40bd-9510-763109a89559" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 10:30:41.478406 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:41.478369    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r" podUID="8db4110e-b735-4859-8364-ab34da572d08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 10:30:42.481049 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:42.481011    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-nctjt" podUID="aff757f2-ce41-40bd-9510-763109a89559" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 10:30:51.478344 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:51.478299    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r" podUID="8db4110e-b735-4859-8364-ab34da572d08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 10:30:52.482809 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:52.482743    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-nctjt" podUID="aff757f2-ce41-40bd-9510-763109a89559" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 10:30:53.229635 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:53.229592    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:30:53.289707 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:30:53.289672    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:31:01.478337 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:31:01.478299    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r" podUID="8db4110e-b735-4859-8364-ab34da572d08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 10:31:02.481845 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:31:02.481806    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-nctjt" podUID="aff757f2-ce41-40bd-9510-763109a89559" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 10:31:11.478611 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:31:11.478575    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r" podUID="8db4110e-b735-4859-8364-ab34da572d08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 10:31:12.481275 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:31:12.481236    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-nctjt" podUID="aff757f2-ce41-40bd-9510-763109a89559" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 10:31:21.478680 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:31:21.478644    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r" podUID="8db4110e-b735-4859-8364-ab34da572d08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 10:31:22.481073 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:31:22.481033    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-nctjt" podUID="aff757f2-ce41-40bd-9510-763109a89559" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 10:31:31.478952 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:31:31.478921    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r"
Jul 28 10:31:32.481937 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:31:32.481908    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-nctjt"
Jul 28 10:31:39.992223 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:31:39.992122    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-nctjt"]
Jul 28 10:31:39.992667 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:31:39.992419    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-nctjt" podUID="aff757f2-ce41-40bd-9510-763109a89559" containerName="kserve-container" containerID="cri-o://f682a7f20c17f36f43d56ea3eccf9e2a5c3ebb90332ead76934fe6ac9abb1b2a" gracePeriod=30
Jul 28 10:31:39.992667 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:31:39.992438    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-nctjt" podUID="aff757f2-ce41-40bd-9510-763109a89559" containerName="kube-rbac-proxy" containerID="cri-o://2c5b4348ede083ac4d3868a43e1f2f0b4ae34ab15861d35a9622dec78dc2311d" gracePeriod=30
Jul 28 10:31:39.994096 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:31:39.994059    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:31:40.875877 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:31:40.875842    2583 generic.go:356] "Generic (PLEG): container finished" podID="aff757f2-ce41-40bd-9510-763109a89559" containerID="2c5b4348ede083ac4d3868a43e1f2f0b4ae34ab15861d35a9622dec78dc2311d" exitCode=2
Jul 28 10:31:40.876048 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:31:40.875915    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-nctjt" event={"ID":"aff757f2-ce41-40bd-9510-763109a89559","Type":"ContainerDied","Data":"2c5b4348ede083ac4d3868a43e1f2f0b4ae34ab15861d35a9622dec78dc2311d"}
Jul 28 10:31:42.477088 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:31:42.477041    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-nctjt" podUID="aff757f2-ce41-40bd-9510-763109a89559" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Jul 28 10:31:42.481575 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:31:42.481530    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-nctjt" podUID="aff757f2-ce41-40bd-9510-763109a89559" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 10:31:44.862800 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:31:44.862771    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-nctjt"
Jul 28 10:31:44.896431 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:31:44.896387    2583 generic.go:356] "Generic (PLEG): container finished" podID="aff757f2-ce41-40bd-9510-763109a89559" containerID="f682a7f20c17f36f43d56ea3eccf9e2a5c3ebb90332ead76934fe6ac9abb1b2a" exitCode=0
Jul 28 10:31:44.896791 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:31:44.896428    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-nctjt" event={"ID":"aff757f2-ce41-40bd-9510-763109a89559","Type":"ContainerDied","Data":"f682a7f20c17f36f43d56ea3eccf9e2a5c3ebb90332ead76934fe6ac9abb1b2a"}
Jul 28 10:31:44.896791 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:31:44.896491    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-nctjt" event={"ID":"aff757f2-ce41-40bd-9510-763109a89559","Type":"ContainerDied","Data":"98af2267bf60add5282e90947c5d193aff1c64c1cd9364187592c7e09ffb65c8"}
Jul 28 10:31:44.896791 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:31:44.896443    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-nctjt"
Jul 28 10:31:44.896791 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:31:44.896520    2583 scope.go:117] "RemoveContainer" containerID="2c5b4348ede083ac4d3868a43e1f2f0b4ae34ab15861d35a9622dec78dc2311d"
Jul 28 10:31:44.906593 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:31:44.906574    2583 scope.go:117] "RemoveContainer" containerID="f682a7f20c17f36f43d56ea3eccf9e2a5c3ebb90332ead76934fe6ac9abb1b2a"
Jul 28 10:31:44.914107 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:31:44.914091    2583 scope.go:117] "RemoveContainer" containerID="548ea479da5eaed794a0404e22f7a9b79c65e59a5136667666e48fdf64527919"
Jul 28 10:31:44.921506 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:31:44.921483    2583 scope.go:117] "RemoveContainer" containerID="2c5b4348ede083ac4d3868a43e1f2f0b4ae34ab15861d35a9622dec78dc2311d"
Jul 28 10:31:44.921767 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:31:44.921744    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c5b4348ede083ac4d3868a43e1f2f0b4ae34ab15861d35a9622dec78dc2311d\": container with ID starting with 2c5b4348ede083ac4d3868a43e1f2f0b4ae34ab15861d35a9622dec78dc2311d not found: ID does not exist" containerID="2c5b4348ede083ac4d3868a43e1f2f0b4ae34ab15861d35a9622dec78dc2311d"
Jul 28 10:31:44.921814 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:31:44.921779    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c5b4348ede083ac4d3868a43e1f2f0b4ae34ab15861d35a9622dec78dc2311d"} err="failed to get container status \"2c5b4348ede083ac4d3868a43e1f2f0b4ae34ab15861d35a9622dec78dc2311d\": rpc error: code = NotFound desc = could not find container \"2c5b4348ede083ac4d3868a43e1f2f0b4ae34ab15861d35a9622dec78dc2311d\": container with ID starting with 2c5b4348ede083ac4d3868a43e1f2f0b4ae34ab15861d35a9622dec78dc2311d not found: ID does not exist"
Jul 28 10:31:44.921814 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:31:44.921804    2583 scope.go:117] "RemoveContainer" containerID="f682a7f20c17f36f43d56ea3eccf9e2a5c3ebb90332ead76934fe6ac9abb1b2a"
Jul 28 10:31:44.922046 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:31:44.922032    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f682a7f20c17f36f43d56ea3eccf9e2a5c3ebb90332ead76934fe6ac9abb1b2a\": container with ID starting with f682a7f20c17f36f43d56ea3eccf9e2a5c3ebb90332ead76934fe6ac9abb1b2a not found: ID does not exist" containerID="f682a7f20c17f36f43d56ea3eccf9e2a5c3ebb90332ead76934fe6ac9abb1b2a"
Jul 28 10:31:44.922091 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:31:44.922051    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f682a7f20c17f36f43d56ea3eccf9e2a5c3ebb90332ead76934fe6ac9abb1b2a"} err="failed to get container status \"f682a7f20c17f36f43d56ea3eccf9e2a5c3ebb90332ead76934fe6ac9abb1b2a\": rpc error: code = NotFound desc = could not find container \"f682a7f20c17f36f43d56ea3eccf9e2a5c3ebb90332ead76934fe6ac9abb1b2a\": container with ID starting with f682a7f20c17f36f43d56ea3eccf9e2a5c3ebb90332ead76934fe6ac9abb1b2a not found: ID does not exist"
Jul 28 10:31:44.922091 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:31:44.922064    2583 scope.go:117] "RemoveContainer" containerID="548ea479da5eaed794a0404e22f7a9b79c65e59a5136667666e48fdf64527919"
Jul 28 10:31:44.922318 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:31:44.922298    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"548ea479da5eaed794a0404e22f7a9b79c65e59a5136667666e48fdf64527919\": container with ID starting with 548ea479da5eaed794a0404e22f7a9b79c65e59a5136667666e48fdf64527919 not found: ID does not exist" containerID="548ea479da5eaed794a0404e22f7a9b79c65e59a5136667666e48fdf64527919"
Jul 28 10:31:44.922373 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:31:44.922324    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"548ea479da5eaed794a0404e22f7a9b79c65e59a5136667666e48fdf64527919"} err="failed to get container status \"548ea479da5eaed794a0404e22f7a9b79c65e59a5136667666e48fdf64527919\": rpc error: code = NotFound desc = could not find container \"548ea479da5eaed794a0404e22f7a9b79c65e59a5136667666e48fdf64527919\": container with ID starting with 548ea479da5eaed794a0404e22f7a9b79c65e59a5136667666e48fdf64527919 not found: ID does not exist"
Jul 28 10:31:44.948828 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:31:44.948798    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aff757f2-ce41-40bd-9510-763109a89559-kserve-provision-location\") pod \"aff757f2-ce41-40bd-9510-763109a89559\" (UID: \"aff757f2-ce41-40bd-9510-763109a89559\") "
Jul 28 10:31:44.948981 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:31:44.948843    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aff757f2-ce41-40bd-9510-763109a89559-proxy-tls\") pod \"aff757f2-ce41-40bd-9510-763109a89559\" (UID: \"aff757f2-ce41-40bd-9510-763109a89559\") "
Jul 28 10:31:44.948981 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:31:44.948871    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d7vq6\" (UniqueName: \"kubernetes.io/projected/aff757f2-ce41-40bd-9510-763109a89559-kube-api-access-d7vq6\") pod \"aff757f2-ce41-40bd-9510-763109a89559\" (UID: \"aff757f2-ce41-40bd-9510-763109a89559\") "
Jul 28 10:31:44.948981 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:31:44.948930    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aff757f2-ce41-40bd-9510-763109a89559-cabundle-cert\") pod \"aff757f2-ce41-40bd-9510-763109a89559\" (UID: \"aff757f2-ce41-40bd-9510-763109a89559\") "
Jul 28 10:31:44.949140 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:31:44.948992    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aff757f2-ce41-40bd-9510-763109a89559-kube-rbac-proxy-sar-config\") pod \"aff757f2-ce41-40bd-9510-763109a89559\" (UID: \"aff757f2-ce41-40bd-9510-763109a89559\") "
Jul 28 10:31:44.949140 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:31:44.949109    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aff757f2-ce41-40bd-9510-763109a89559-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "aff757f2-ce41-40bd-9510-763109a89559" (UID: "aff757f2-ce41-40bd-9510-763109a89559"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:31:44.949272 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:31:44.949257    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aff757f2-ce41-40bd-9510-763109a89559-kserve-provision-location\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:31:44.949380 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:31:44.949356    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aff757f2-ce41-40bd-9510-763109a89559-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "aff757f2-ce41-40bd-9510-763109a89559" (UID: "aff757f2-ce41-40bd-9510-763109a89559"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:31:44.949434 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:31:44.949385    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aff757f2-ce41-40bd-9510-763109a89559-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "aff757f2-ce41-40bd-9510-763109a89559" (UID: "aff757f2-ce41-40bd-9510-763109a89559"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:31:44.951170 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:31:44.951145    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aff757f2-ce41-40bd-9510-763109a89559-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "aff757f2-ce41-40bd-9510-763109a89559" (UID: "aff757f2-ce41-40bd-9510-763109a89559"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:31:44.951276 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:31:44.951214    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aff757f2-ce41-40bd-9510-763109a89559-kube-api-access-d7vq6" (OuterVolumeSpecName: "kube-api-access-d7vq6") pod "aff757f2-ce41-40bd-9510-763109a89559" (UID: "aff757f2-ce41-40bd-9510-763109a89559"). InnerVolumeSpecName "kube-api-access-d7vq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:31:45.049800 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:31:45.049747    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aff757f2-ce41-40bd-9510-763109a89559-proxy-tls\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:31:45.049800 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:31:45.049798    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7vq6\" (UniqueName: \"kubernetes.io/projected/aff757f2-ce41-40bd-9510-763109a89559-kube-api-access-d7vq6\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:31:45.049800 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:31:45.049811    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aff757f2-ce41-40bd-9510-763109a89559-cabundle-cert\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:31:45.050051 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:31:45.049824    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aff757f2-ce41-40bd-9510-763109a89559-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:31:45.221724 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:31:45.221687    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-nctjt"]
Jul 28 10:31:45.225910 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:31:45.225883    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-nctjt"]
Jul 28 10:31:45.437651 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:31:45.437608    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aff757f2-ce41-40bd-9510-763109a89559" path="/var/lib/kubelet/pods/aff757f2-ce41-40bd-9510-763109a89559/volumes"
Jul 28 10:33:30.223431 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:30.223399    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r"]
Jul 28 10:33:30.225969 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:30.223646    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r" podUID="8db4110e-b735-4859-8364-ab34da572d08" containerName="kserve-container" containerID="cri-o://bacd9265b7ee67025a079b68bd816dfad047c9e4dd7a65cabedc3f176ffa93bb" gracePeriod=30
Jul 28 10:33:30.225969 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:30.223653    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r" podUID="8db4110e-b735-4859-8364-ab34da572d08" containerName="kube-rbac-proxy" containerID="cri-o://ca8f84e5e8efa2cf691b47c0b31841ec642240f5def2e2f7f50db0be27c2a7c5" gracePeriod=30
Jul 28 10:33:30.360538 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:30.360508    2583 generic.go:356] "Generic (PLEG): container finished" podID="8db4110e-b735-4859-8364-ab34da572d08" containerID="ca8f84e5e8efa2cf691b47c0b31841ec642240f5def2e2f7f50db0be27c2a7c5" exitCode=2
Jul 28 10:33:30.360671 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:30.360568    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r" event={"ID":"8db4110e-b735-4859-8364-ab34da572d08","Type":"ContainerDied","Data":"ca8f84e5e8efa2cf691b47c0b31841ec642240f5def2e2f7f50db0be27c2a7c5"}
Jul 28 10:33:31.206671 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:31.206634    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:33:31.472951 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:31.472861    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r" podUID="8db4110e-b735-4859-8364-ab34da572d08" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Jul 28 10:33:31.478082 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:31.478051    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r" podUID="8db4110e-b735-4859-8364-ab34da572d08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 10:33:34.563794 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:34.563771    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r"
Jul 28 10:33:34.646766 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:34.646742    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8db4110e-b735-4859-8364-ab34da572d08-kube-rbac-proxy-sar-config\") pod \"8db4110e-b735-4859-8364-ab34da572d08\" (UID: \"8db4110e-b735-4859-8364-ab34da572d08\") "
Jul 28 10:33:34.646894 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:34.646772    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8db4110e-b735-4859-8364-ab34da572d08-cabundle-cert\") pod \"8db4110e-b735-4859-8364-ab34da572d08\" (UID: \"8db4110e-b735-4859-8364-ab34da572d08\") "
Jul 28 10:33:34.646894 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:34.646798    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8db4110e-b735-4859-8364-ab34da572d08-proxy-tls\") pod \"8db4110e-b735-4859-8364-ab34da572d08\" (UID: \"8db4110e-b735-4859-8364-ab34da572d08\") "
Jul 28 10:33:34.646967 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:34.646925    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8db4110e-b735-4859-8364-ab34da572d08-kserve-provision-location\") pod \"8db4110e-b735-4859-8364-ab34da572d08\" (UID: \"8db4110e-b735-4859-8364-ab34da572d08\") "
Jul 28 10:33:34.647021 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:34.647004    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j8ckg\" (UniqueName: \"kubernetes.io/projected/8db4110e-b735-4859-8364-ab34da572d08-kube-api-access-j8ckg\") pod \"8db4110e-b735-4859-8364-ab34da572d08\" (UID: \"8db4110e-b735-4859-8364-ab34da572d08\") "
Jul 28 10:33:34.647273 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:34.647243    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8db4110e-b735-4859-8364-ab34da572d08-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8db4110e-b735-4859-8364-ab34da572d08" (UID: "8db4110e-b735-4859-8364-ab34da572d08"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:33:34.647273 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:34.647253    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8db4110e-b735-4859-8364-ab34da572d08-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8db4110e-b735-4859-8364-ab34da572d08" (UID: "8db4110e-b735-4859-8364-ab34da572d08"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:33:34.647421 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:34.647276    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8db4110e-b735-4859-8364-ab34da572d08-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8db4110e-b735-4859-8364-ab34da572d08" (UID: "8db4110e-b735-4859-8364-ab34da572d08"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:33:34.648909 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:34.648887    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8db4110e-b735-4859-8364-ab34da572d08-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8db4110e-b735-4859-8364-ab34da572d08" (UID: "8db4110e-b735-4859-8364-ab34da572d08"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:33:34.649001 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:34.648919    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8db4110e-b735-4859-8364-ab34da572d08-kube-api-access-j8ckg" (OuterVolumeSpecName: "kube-api-access-j8ckg") pod "8db4110e-b735-4859-8364-ab34da572d08" (UID: "8db4110e-b735-4859-8364-ab34da572d08"). InnerVolumeSpecName "kube-api-access-j8ckg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:33:34.748368 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:34.748344    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8ckg\" (UniqueName: \"kubernetes.io/projected/8db4110e-b735-4859-8364-ab34da572d08-kube-api-access-j8ckg\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:33:34.748368 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:34.748366    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8db4110e-b735-4859-8364-ab34da572d08-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:33:34.748496 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:34.748377    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8db4110e-b735-4859-8364-ab34da572d08-cabundle-cert\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:33:34.748496 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:34.748387    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8db4110e-b735-4859-8364-ab34da572d08-proxy-tls\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:33:34.748496 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:34.748395    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8db4110e-b735-4859-8364-ab34da572d08-kserve-provision-location\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:33:35.314932 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:35.314902    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:33:35.381311 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:35.381275    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:33:35.385966 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:35.385939    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-wglnd"]
Jul 28 10:33:35.386330 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:35.386306    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aff757f2-ce41-40bd-9510-763109a89559" containerName="storage-initializer"
Jul 28 10:33:35.386330 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:35.386324    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="aff757f2-ce41-40bd-9510-763109a89559" containerName="storage-initializer"
Jul 28 10:33:35.386494 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:35.386341    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8db4110e-b735-4859-8364-ab34da572d08" containerName="storage-initializer"
Jul 28 10:33:35.386494 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:35.386348    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8db4110e-b735-4859-8364-ab34da572d08" containerName="storage-initializer"
Jul 28 10:33:35.386494 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:35.386358    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aff757f2-ce41-40bd-9510-763109a89559" containerName="kserve-container"
Jul 28 10:33:35.386494 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:35.386364    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="aff757f2-ce41-40bd-9510-763109a89559" containerName="kserve-container"
Jul 28 10:33:35.386494 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:35.386370    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8db4110e-b735-4859-8364-ab34da572d08" containerName="kserve-container"
Jul 28 10:33:35.386494 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:35.386375    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8db4110e-b735-4859-8364-ab34da572d08" containerName="kserve-container"
Jul 28 10:33:35.386494 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:35.386388    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aff757f2-ce41-40bd-9510-763109a89559" containerName="kube-rbac-proxy"
Jul 28 10:33:35.386494 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:35.386394    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="aff757f2-ce41-40bd-9510-763109a89559" containerName="kube-rbac-proxy"
Jul 28 10:33:35.386494 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:35.386402    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8db4110e-b735-4859-8364-ab34da572d08" containerName="kube-rbac-proxy"
Jul 28 10:33:35.386494 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:35.386411    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8db4110e-b735-4859-8364-ab34da572d08" containerName="kube-rbac-proxy"
Jul 28 10:33:35.386494 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:35.386475    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="aff757f2-ce41-40bd-9510-763109a89559" containerName="kserve-container"
Jul 28 10:33:35.386494 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:35.386486    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="aff757f2-ce41-40bd-9510-763109a89559" containerName="kube-rbac-proxy"
Jul 28 10:33:35.386494 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:35.386496    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8db4110e-b735-4859-8364-ab34da572d08" containerName="kserve-container"
Jul 28 10:33:35.387054 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:35.386507    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8db4110e-b735-4859-8364-ab34da572d08" containerName="kube-rbac-proxy"
Jul 28 10:33:35.387531 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:35.387500    2583 generic.go:356] "Generic (PLEG): container finished" podID="8db4110e-b735-4859-8364-ab34da572d08" containerID="bacd9265b7ee67025a079b68bd816dfad047c9e4dd7a65cabedc3f176ffa93bb" exitCode=0
Jul 28 10:33:35.387671 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:35.387659    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r"
Jul 28 10:33:35.390019 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:35.389998    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r" event={"ID":"8db4110e-b735-4859-8364-ab34da572d08","Type":"ContainerDied","Data":"bacd9265b7ee67025a079b68bd816dfad047c9e4dd7a65cabedc3f176ffa93bb"}
Jul 28 10:33:35.390116 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:35.390037    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r" event={"ID":"8db4110e-b735-4859-8364-ab34da572d08","Type":"ContainerDied","Data":"557d8d0c6781fab835ec2150720f9428f39f98cb95cb948f0069d2b7507880c9"}
Jul 28 10:33:35.390164 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:35.390120    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-wglnd"
Jul 28 10:33:35.390164 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:35.390126    2583 scope.go:117] "RemoveContainer" containerID="ca8f84e5e8efa2cf691b47c0b31841ec642240f5def2e2f7f50db0be27c2a7c5"
Jul 28 10:33:35.392155 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:35.392124    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Jul 28 10:33:35.392750 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:35.392589    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-d7gr5\""
Jul 28 10:33:35.392750 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:35.392605    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 10:33:35.392908 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:35.392812    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 10:33:35.393000 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:35.392984    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 10:33:35.393073 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:35.393013    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Jul 28 10:33:35.393147 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:35.393131    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 10:33:35.401082 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:35.401002    2583 scope.go:117] "RemoveContainer" containerID="bacd9265b7ee67025a079b68bd816dfad047c9e4dd7a65cabedc3f176ffa93bb"
Jul 28 10:33:35.404444 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:35.404420    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-wglnd"]
Jul 28 10:33:35.410433 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:35.410418    2583 scope.go:117] "RemoveContainer" containerID="99111529909d5139873b5d5c5c3e0fc97a13d12edd6c7e3e3a44169a85a5d210"
Jul 28 10:33:35.419017 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:35.418983    2583 scope.go:117] "RemoveContainer" containerID="ca8f84e5e8efa2cf691b47c0b31841ec642240f5def2e2f7f50db0be27c2a7c5"
Jul 28 10:33:35.419298 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:33:35.419278    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca8f84e5e8efa2cf691b47c0b31841ec642240f5def2e2f7f50db0be27c2a7c5\": container with ID starting with ca8f84e5e8efa2cf691b47c0b31841ec642240f5def2e2f7f50db0be27c2a7c5 not found: ID does not exist" containerID="ca8f84e5e8efa2cf691b47c0b31841ec642240f5def2e2f7f50db0be27c2a7c5"
Jul 28 10:33:35.419382 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:35.419304    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca8f84e5e8efa2cf691b47c0b31841ec642240f5def2e2f7f50db0be27c2a7c5"} err="failed to get container status \"ca8f84e5e8efa2cf691b47c0b31841ec642240f5def2e2f7f50db0be27c2a7c5\": rpc error: code = NotFound desc = could not find container \"ca8f84e5e8efa2cf691b47c0b31841ec642240f5def2e2f7f50db0be27c2a7c5\": container with ID starting with ca8f84e5e8efa2cf691b47c0b31841ec642240f5def2e2f7f50db0be27c2a7c5 not found: ID does not exist"
Jul 28 10:33:35.419382 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:35.419322    2583 scope.go:117] "RemoveContainer" containerID="bacd9265b7ee67025a079b68bd816dfad047c9e4dd7a65cabedc3f176ffa93bb"
Jul 28 10:33:35.419582 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:33:35.419567    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bacd9265b7ee67025a079b68bd816dfad047c9e4dd7a65cabedc3f176ffa93bb\": container with ID starting with bacd9265b7ee67025a079b68bd816dfad047c9e4dd7a65cabedc3f176ffa93bb not found: ID does not exist" containerID="bacd9265b7ee67025a079b68bd816dfad047c9e4dd7a65cabedc3f176ffa93bb"
Jul 28 10:33:35.419637 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:35.419588    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bacd9265b7ee67025a079b68bd816dfad047c9e4dd7a65cabedc3f176ffa93bb"} err="failed to get container status \"bacd9265b7ee67025a079b68bd816dfad047c9e4dd7a65cabedc3f176ffa93bb\": rpc error: code = NotFound desc = could not find container \"bacd9265b7ee67025a079b68bd816dfad047c9e4dd7a65cabedc3f176ffa93bb\": container with ID starting with bacd9265b7ee67025a079b68bd816dfad047c9e4dd7a65cabedc3f176ffa93bb not found: ID does not exist"
Jul 28 10:33:35.419637 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:35.419603    2583 scope.go:117] "RemoveContainer" containerID="99111529909d5139873b5d5c5c3e0fc97a13d12edd6c7e3e3a44169a85a5d210"
Jul 28 10:33:35.419861 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:33:35.419845    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99111529909d5139873b5d5c5c3e0fc97a13d12edd6c7e3e3a44169a85a5d210\": container with ID starting with 99111529909d5139873b5d5c5c3e0fc97a13d12edd6c7e3e3a44169a85a5d210 not found: ID does not exist" containerID="99111529909d5139873b5d5c5c3e0fc97a13d12edd6c7e3e3a44169a85a5d210"
Jul 28 10:33:35.419919 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:35.419875    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99111529909d5139873b5d5c5c3e0fc97a13d12edd6c7e3e3a44169a85a5d210"} err="failed to get container status \"99111529909d5139873b5d5c5c3e0fc97a13d12edd6c7e3e3a44169a85a5d210\": rpc error: code = NotFound desc = could not find container \"99111529909d5139873b5d5c5c3e0fc97a13d12edd6c7e3e3a44169a85a5d210\": container with ID starting with 99111529909d5139873b5d5c5c3e0fc97a13d12edd6c7e3e3a44169a85a5d210 not found: ID does not exist"
Jul 28 10:33:35.432828 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:35.432805    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r"]
Jul 28 10:33:35.439950 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:35.439931    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-v5n6r"]
Jul 28 10:33:35.452764 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:35.452739    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2fmh\" (UniqueName: \"kubernetes.io/projected/92dda428-ac3b-4f25-8a19-143b08090311-kube-api-access-s2fmh\") pod \"isvc-canary-rollback-predictor-bb95f9bbd-wglnd\" (UID: \"92dda428-ac3b-4f25-8a19-143b08090311\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-wglnd"
Jul 28 10:33:35.452865 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:35.452790    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/92dda428-ac3b-4f25-8a19-143b08090311-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-bb95f9bbd-wglnd\" (UID: \"92dda428-ac3b-4f25-8a19-143b08090311\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-wglnd"
Jul 28 10:33:35.452865 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:35.452841    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92dda428-ac3b-4f25-8a19-143b08090311-proxy-tls\") pod \"isvc-canary-rollback-predictor-bb95f9bbd-wglnd\" (UID: \"92dda428-ac3b-4f25-8a19-143b08090311\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-wglnd"
Jul 28 10:33:35.452983 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:35.452866    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/92dda428-ac3b-4f25-8a19-143b08090311-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-bb95f9bbd-wglnd\" (UID: \"92dda428-ac3b-4f25-8a19-143b08090311\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-wglnd"
Jul 28 10:33:35.452983 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:35.452905    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/92dda428-ac3b-4f25-8a19-143b08090311-cabundle-cert\") pod \"isvc-canary-rollback-predictor-bb95f9bbd-wglnd\" (UID: \"92dda428-ac3b-4f25-8a19-143b08090311\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-wglnd"
Jul 28 10:33:35.554222 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:35.554175    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2fmh\" (UniqueName: \"kubernetes.io/projected/92dda428-ac3b-4f25-8a19-143b08090311-kube-api-access-s2fmh\") pod \"isvc-canary-rollback-predictor-bb95f9bbd-wglnd\" (UID: \"92dda428-ac3b-4f25-8a19-143b08090311\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-wglnd"
Jul 28 10:33:35.554329 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:35.554262    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/92dda428-ac3b-4f25-8a19-143b08090311-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-bb95f9bbd-wglnd\" (UID: \"92dda428-ac3b-4f25-8a19-143b08090311\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-wglnd"
Jul 28 10:33:35.554329 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:35.554300    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92dda428-ac3b-4f25-8a19-143b08090311-proxy-tls\") pod \"isvc-canary-rollback-predictor-bb95f9bbd-wglnd\" (UID: \"92dda428-ac3b-4f25-8a19-143b08090311\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-wglnd"
Jul 28 10:33:35.554531 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:35.554326    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/92dda428-ac3b-4f25-8a19-143b08090311-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-bb95f9bbd-wglnd\" (UID: \"92dda428-ac3b-4f25-8a19-143b08090311\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-wglnd"
Jul 28 10:33:35.554531 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:35.554389    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/92dda428-ac3b-4f25-8a19-143b08090311-cabundle-cert\") pod \"isvc-canary-rollback-predictor-bb95f9bbd-wglnd\" (UID: \"92dda428-ac3b-4f25-8a19-143b08090311\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-wglnd"
Jul 28 10:33:35.555147 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:35.554686    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/92dda428-ac3b-4f25-8a19-143b08090311-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-bb95f9bbd-wglnd\" (UID: \"92dda428-ac3b-4f25-8a19-143b08090311\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-wglnd"
Jul 28 10:33:35.555147 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:33:35.554787    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-predictor-serving-cert: secret "isvc-canary-rollback-predictor-serving-cert" not found
Jul 28 10:33:35.555147 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:33:35.554853    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92dda428-ac3b-4f25-8a19-143b08090311-proxy-tls podName:92dda428-ac3b-4f25-8a19-143b08090311 nodeName:}" failed. No retries permitted until 2026-07-28 10:33:36.054833207 +0000 UTC m=+1555.155339746 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/92dda428-ac3b-4f25-8a19-143b08090311-proxy-tls") pod "isvc-canary-rollback-predictor-bb95f9bbd-wglnd" (UID: "92dda428-ac3b-4f25-8a19-143b08090311") : secret "isvc-canary-rollback-predictor-serving-cert" not found
Jul 28 10:33:35.555348 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:35.555212    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/92dda428-ac3b-4f25-8a19-143b08090311-cabundle-cert\") pod \"isvc-canary-rollback-predictor-bb95f9bbd-wglnd\" (UID: \"92dda428-ac3b-4f25-8a19-143b08090311\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-wglnd"
Jul 28 10:33:35.555348 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:35.555339    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/92dda428-ac3b-4f25-8a19-143b08090311-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-bb95f9bbd-wglnd\" (UID: \"92dda428-ac3b-4f25-8a19-143b08090311\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-wglnd"
Jul 28 10:33:35.562964 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:35.562941    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2fmh\" (UniqueName: \"kubernetes.io/projected/92dda428-ac3b-4f25-8a19-143b08090311-kube-api-access-s2fmh\") pod \"isvc-canary-rollback-predictor-bb95f9bbd-wglnd\" (UID: \"92dda428-ac3b-4f25-8a19-143b08090311\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-wglnd"
Jul 28 10:33:35.640130 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:35.640074    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:33:36.060019 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:36.059989    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92dda428-ac3b-4f25-8a19-143b08090311-proxy-tls\") pod \"isvc-canary-rollback-predictor-bb95f9bbd-wglnd\" (UID: \"92dda428-ac3b-4f25-8a19-143b08090311\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-wglnd"
Jul 28 10:33:36.060211 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:33:36.060148    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-predictor-serving-cert: secret "isvc-canary-rollback-predictor-serving-cert" not found
Jul 28 10:33:36.060259 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:33:36.060243    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92dda428-ac3b-4f25-8a19-143b08090311-proxy-tls podName:92dda428-ac3b-4f25-8a19-143b08090311 nodeName:}" failed. No retries permitted until 2026-07-28 10:33:37.060227596 +0000 UTC m=+1556.160734129 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/92dda428-ac3b-4f25-8a19-143b08090311-proxy-tls") pod "isvc-canary-rollback-predictor-bb95f9bbd-wglnd" (UID: "92dda428-ac3b-4f25-8a19-143b08090311") : secret "isvc-canary-rollback-predictor-serving-cert" not found
Jul 28 10:33:36.213527 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:36.213495    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:33:37.066752 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:37.066719    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92dda428-ac3b-4f25-8a19-143b08090311-proxy-tls\") pod \"isvc-canary-rollback-predictor-bb95f9bbd-wglnd\" (UID: \"92dda428-ac3b-4f25-8a19-143b08090311\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-wglnd"
Jul 28 10:33:37.069040 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:37.069018    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92dda428-ac3b-4f25-8a19-143b08090311-proxy-tls\") pod \"isvc-canary-rollback-predictor-bb95f9bbd-wglnd\" (UID: \"92dda428-ac3b-4f25-8a19-143b08090311\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-wglnd"
Jul 28 10:33:37.205544 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:37.205513    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-wglnd"
Jul 28 10:33:37.334111 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:37.334085    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-wglnd"]
Jul 28 10:33:37.335893 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:33:37.335864    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92dda428_ac3b_4f25_8a19_143b08090311.slice/crio-24f83e52ab0b3a23f2cda52991d8751f2365beffcbb0496818fde738b5a68dbe WatchSource:0}: Error finding container 24f83e52ab0b3a23f2cda52991d8751f2365beffcbb0496818fde738b5a68dbe: Status 404 returned error can't find the container with id 24f83e52ab0b3a23f2cda52991d8751f2365beffcbb0496818fde738b5a68dbe
Jul 28 10:33:37.398507 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:37.398473    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-wglnd" event={"ID":"92dda428-ac3b-4f25-8a19-143b08090311","Type":"ContainerStarted","Data":"3959245e2163460d16993bb9690dfdfd04b647f34e720202a2129ee5dca401a0"}
Jul 28 10:33:37.398605 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:37.398569    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-wglnd" event={"ID":"92dda428-ac3b-4f25-8a19-143b08090311","Type":"ContainerStarted","Data":"24f83e52ab0b3a23f2cda52991d8751f2365beffcbb0496818fde738b5a68dbe"}
Jul 28 10:33:37.438455 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:37.438428    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8db4110e-b735-4859-8364-ab34da572d08" path="/var/lib/kubelet/pods/8db4110e-b735-4859-8364-ab34da572d08/volumes"
Jul 28 10:33:41.418342 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:41.418306    2583 generic.go:356] "Generic (PLEG): container finished" podID="92dda428-ac3b-4f25-8a19-143b08090311" containerID="3959245e2163460d16993bb9690dfdfd04b647f34e720202a2129ee5dca401a0" exitCode=0
Jul 28 10:33:41.418679 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:41.418355    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-wglnd" event={"ID":"92dda428-ac3b-4f25-8a19-143b08090311","Type":"ContainerDied","Data":"3959245e2163460d16993bb9690dfdfd04b647f34e720202a2129ee5dca401a0"}
Jul 28 10:33:42.431264 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:42.431226    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-wglnd" event={"ID":"92dda428-ac3b-4f25-8a19-143b08090311","Type":"ContainerStarted","Data":"6e6f275bf70912d61bb2eba0f18cfbf4d5c2920861c686a8f256c43e6e9b71c9"}
Jul 28 10:33:42.431755 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:42.431333    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-wglnd" event={"ID":"92dda428-ac3b-4f25-8a19-143b08090311","Type":"ContainerStarted","Data":"0144d981414888a459b61088b1397c95c1b60099b8293017c4e8c4cfae39a1b2"}
Jul 28 10:33:42.431755 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:42.431557    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-wglnd"
Jul 28 10:33:42.452101 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:42.452053    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-wglnd" podStartSLOduration=3.739308495 podStartE2EDuration="7.452039572s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.712731077s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:33:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:33:42.449704451 +0000 UTC m=+1561.550211030" watchObservedRunningTime="2026-07-28 10:33:42.452039572 +0000 UTC m=+1561.552546125"
Jul 28 10:33:43.438240 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:43.438204    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-wglnd" podUID="92dda428-ac3b-4f25-8a19-143b08090311" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 10:33:43.438697 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:43.438504    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-wglnd"
Jul 28 10:33:44.441637 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:44.441596    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-wglnd" podUID="92dda428-ac3b-4f25-8a19-143b08090311" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 10:33:49.447041 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:49.447011    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-wglnd"
Jul 28 10:33:49.447530 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:49.447504    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-wglnd" podUID="92dda428-ac3b-4f25-8a19-143b08090311" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 10:33:59.448148 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:33:59.448111    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-wglnd" podUID="92dda428-ac3b-4f25-8a19-143b08090311" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 10:34:09.447925 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:34:09.447887    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-wglnd" podUID="92dda428-ac3b-4f25-8a19-143b08090311" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 10:34:19.448233 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:34:19.448170    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-wglnd" podUID="92dda428-ac3b-4f25-8a19-143b08090311" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 10:34:29.447769 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:34:29.447730    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-wglnd" podUID="92dda428-ac3b-4f25-8a19-143b08090311" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 10:34:30.099453 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:34:30.099420    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:34:39.447726 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:34:39.447639    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-wglnd" podUID="92dda428-ac3b-4f25-8a19-143b08090311" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 10:34:49.448337 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:34:49.448295    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-wglnd" podUID="92dda428-ac3b-4f25-8a19-143b08090311" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 10:34:59.448934 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:34:59.448902    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-wglnd"
Jul 28 10:35:05.437116 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:35:05.437079    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:36:35.613263 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:35.613226    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-wglnd"]
Jul 28 10:36:35.613803 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:35.613468    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-wglnd" podUID="92dda428-ac3b-4f25-8a19-143b08090311" containerName="kserve-container" containerID="cri-o://0144d981414888a459b61088b1397c95c1b60099b8293017c4e8c4cfae39a1b2" gracePeriod=30
Jul 28 10:36:35.613803 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:35.613517    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-wglnd" podUID="92dda428-ac3b-4f25-8a19-143b08090311" containerName="kube-rbac-proxy" containerID="cri-o://6e6f275bf70912d61bb2eba0f18cfbf4d5c2920861c686a8f256c43e6e9b71c9" gracePeriod=30
Jul 28 10:36:35.654652 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:35.654617    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:36:36.192859 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:36.192826    2583 generic.go:356] "Generic (PLEG): container finished" podID="92dda428-ac3b-4f25-8a19-143b08090311" containerID="6e6f275bf70912d61bb2eba0f18cfbf4d5c2920861c686a8f256c43e6e9b71c9" exitCode=2
Jul 28 10:36:36.193035 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:36.192872    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-wglnd" event={"ID":"92dda428-ac3b-4f25-8a19-143b08090311","Type":"ContainerDied","Data":"6e6f275bf70912d61bb2eba0f18cfbf4d5c2920861c686a8f256c43e6e9b71c9"}
Jul 28 10:36:39.442706 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:39.442663    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-wglnd" podUID="92dda428-ac3b-4f25-8a19-143b08090311" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
Jul 28 10:36:39.448081 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:39.448056    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-wglnd" podUID="92dda428-ac3b-4f25-8a19-143b08090311" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 10:36:40.461619 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:40.461594    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-wglnd"
Jul 28 10:36:40.578630 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:40.578537    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/92dda428-ac3b-4f25-8a19-143b08090311-kserve-provision-location\") pod \"92dda428-ac3b-4f25-8a19-143b08090311\" (UID: \"92dda428-ac3b-4f25-8a19-143b08090311\") "
Jul 28 10:36:40.578630 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:40.578582    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92dda428-ac3b-4f25-8a19-143b08090311-proxy-tls\") pod \"92dda428-ac3b-4f25-8a19-143b08090311\" (UID: \"92dda428-ac3b-4f25-8a19-143b08090311\") "
Jul 28 10:36:40.578630 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:40.578620    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s2fmh\" (UniqueName: \"kubernetes.io/projected/92dda428-ac3b-4f25-8a19-143b08090311-kube-api-access-s2fmh\") pod \"92dda428-ac3b-4f25-8a19-143b08090311\" (UID: \"92dda428-ac3b-4f25-8a19-143b08090311\") "
Jul 28 10:36:40.578860 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:40.578651    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/92dda428-ac3b-4f25-8a19-143b08090311-kube-rbac-proxy-sar-config\") pod \"92dda428-ac3b-4f25-8a19-143b08090311\" (UID: \"92dda428-ac3b-4f25-8a19-143b08090311\") "
Jul 28 10:36:40.578860 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:40.578718    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/92dda428-ac3b-4f25-8a19-143b08090311-cabundle-cert\") pod \"92dda428-ac3b-4f25-8a19-143b08090311\" (UID: \"92dda428-ac3b-4f25-8a19-143b08090311\") "
Jul 28 10:36:40.578974 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:40.578929    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92dda428-ac3b-4f25-8a19-143b08090311-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "92dda428-ac3b-4f25-8a19-143b08090311" (UID: "92dda428-ac3b-4f25-8a19-143b08090311"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:36:40.579146 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:40.579118    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92dda428-ac3b-4f25-8a19-143b08090311-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "92dda428-ac3b-4f25-8a19-143b08090311" (UID: "92dda428-ac3b-4f25-8a19-143b08090311"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:36:40.579253 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:40.579123    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92dda428-ac3b-4f25-8a19-143b08090311-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "92dda428-ac3b-4f25-8a19-143b08090311" (UID: "92dda428-ac3b-4f25-8a19-143b08090311"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:36:40.580767 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:40.580746    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92dda428-ac3b-4f25-8a19-143b08090311-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "92dda428-ac3b-4f25-8a19-143b08090311" (UID: "92dda428-ac3b-4f25-8a19-143b08090311"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:36:40.581040 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:40.581026    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92dda428-ac3b-4f25-8a19-143b08090311-kube-api-access-s2fmh" (OuterVolumeSpecName: "kube-api-access-s2fmh") pod "92dda428-ac3b-4f25-8a19-143b08090311" (UID: "92dda428-ac3b-4f25-8a19-143b08090311"). InnerVolumeSpecName "kube-api-access-s2fmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:36:40.680355 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:40.680323    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/92dda428-ac3b-4f25-8a19-143b08090311-kserve-provision-location\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:36:40.680355 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:40.680353    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92dda428-ac3b-4f25-8a19-143b08090311-proxy-tls\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:36:40.680553 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:40.680370    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2fmh\" (UniqueName: \"kubernetes.io/projected/92dda428-ac3b-4f25-8a19-143b08090311-kube-api-access-s2fmh\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:36:40.680553 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:40.680382    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/92dda428-ac3b-4f25-8a19-143b08090311-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:36:40.680553 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:40.680394    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/92dda428-ac3b-4f25-8a19-143b08090311-cabundle-cert\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:36:40.693159 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:40.693124    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:36:40.752957 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:40.752927    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-925g5"]
Jul 28 10:36:40.753292 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:40.753278    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92dda428-ac3b-4f25-8a19-143b08090311" containerName="kube-rbac-proxy"
Jul 28 10:36:40.753344 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:40.753296    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="92dda428-ac3b-4f25-8a19-143b08090311" containerName="kube-rbac-proxy"
Jul 28 10:36:40.753344 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:40.753314    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92dda428-ac3b-4f25-8a19-143b08090311" containerName="storage-initializer"
Jul 28 10:36:40.753344 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:40.753319    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="92dda428-ac3b-4f25-8a19-143b08090311" containerName="storage-initializer"
Jul 28 10:36:40.753344 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:40.753326    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92dda428-ac3b-4f25-8a19-143b08090311" containerName="kserve-container"
Jul 28 10:36:40.753344 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:40.753331    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="92dda428-ac3b-4f25-8a19-143b08090311" containerName="kserve-container"
Jul 28 10:36:40.753508 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:40.753390    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="92dda428-ac3b-4f25-8a19-143b08090311" containerName="kserve-container"
Jul 28 10:36:40.753508 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:40.753399    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="92dda428-ac3b-4f25-8a19-143b08090311" containerName="kube-rbac-proxy"
Jul 28 10:36:40.756719 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:40.756703    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-925g5"
Jul 28 10:36:40.759816 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:40.759792    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd"]
Jul 28 10:36:40.762024 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:40.761994    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Jul 28 10:36:40.762151 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:40.761994    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Jul 28 10:36:40.762371 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:40.762348    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:36:40.762941 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:40.762927    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd"
Jul 28 10:36:40.765388 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:40.765369    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Jul 28 10:36:40.769849 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:40.769826    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-925g5"]
Jul 28 10:36:40.779151 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:40.779128    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd"]
Jul 28 10:36:40.856655 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:40.856561    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:36:40.881601 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:40.881571    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86cff314-e9af-4027-91d9-5df3f7d4ac63-proxy-tls\") pod \"isvc-canary-stop-predictor-cf655cf76-925g5\" (UID: \"86cff314-e9af-4027-91d9-5df3f7d4ac63\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-925g5"
Jul 28 10:36:40.881763 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:40.881622    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86cff314-e9af-4027-91d9-5df3f7d4ac63-kserve-provision-location\") pod \"isvc-canary-stop-predictor-cf655cf76-925g5\" (UID: \"86cff314-e9af-4027-91d9-5df3f7d4ac63\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-925g5"
Jul 28 10:36:40.881763 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:40.881652    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86cff314-e9af-4027-91d9-5df3f7d4ac63-cabundle-cert\") pod \"isvc-canary-stop-predictor-cf655cf76-925g5\" (UID: \"86cff314-e9af-4027-91d9-5df3f7d4ac63\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-925g5"
Jul 28 10:36:40.881763 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:40.881707    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2bl7\" (UniqueName: \"kubernetes.io/projected/4fadb75f-28ee-409b-9c10-f4e16d1b7f10-kube-api-access-l2bl7\") pod \"isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd\" (UID: \"4fadb75f-28ee-409b-9c10-f4e16d1b7f10\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd"
Jul 28 10:36:40.881763 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:40.881735    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4fadb75f-28ee-409b-9c10-f4e16d1b7f10-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd\" (UID: \"4fadb75f-28ee-409b-9c10-f4e16d1b7f10\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd"
Jul 28 10:36:40.881949 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:40.881806    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4fadb75f-28ee-409b-9c10-f4e16d1b7f10-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd\" (UID: \"4fadb75f-28ee-409b-9c10-f4e16d1b7f10\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd"
Jul 28 10:36:40.881949 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:40.881844    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86cff314-e9af-4027-91d9-5df3f7d4ac63-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-cf655cf76-925g5\" (UID: \"86cff314-e9af-4027-91d9-5df3f7d4ac63\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-925g5"
Jul 28 10:36:40.881949 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:40.881862    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqd5b\" (UniqueName: \"kubernetes.io/projected/86cff314-e9af-4027-91d9-5df3f7d4ac63-kube-api-access-sqd5b\") pod \"isvc-canary-stop-predictor-cf655cf76-925g5\" (UID: \"86cff314-e9af-4027-91d9-5df3f7d4ac63\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-925g5"
Jul 28 10:36:40.881949 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:40.881896    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4fadb75f-28ee-409b-9c10-f4e16d1b7f10-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd\" (UID: \"4fadb75f-28ee-409b-9c10-f4e16d1b7f10\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd"
Jul 28 10:36:40.881949 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:40.881915    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4fadb75f-28ee-409b-9c10-f4e16d1b7f10-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd\" (UID: \"4fadb75f-28ee-409b-9c10-f4e16d1b7f10\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd"
Jul 28 10:36:40.957975 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:40.957943    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:36:40.983049 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:40.983021    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4fadb75f-28ee-409b-9c10-f4e16d1b7f10-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd\" (UID: \"4fadb75f-28ee-409b-9c10-f4e16d1b7f10\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd"
Jul 28 10:36:40.983211 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:40.983069    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4fadb75f-28ee-409b-9c10-f4e16d1b7f10-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd\" (UID: \"4fadb75f-28ee-409b-9c10-f4e16d1b7f10\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd"
Jul 28 10:36:40.983271 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:40.983202    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86cff314-e9af-4027-91d9-5df3f7d4ac63-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-cf655cf76-925g5\" (UID: \"86cff314-e9af-4027-91d9-5df3f7d4ac63\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-925g5"
Jul 28 10:36:40.983271 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:40.983241    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqd5b\" (UniqueName: \"kubernetes.io/projected/86cff314-e9af-4027-91d9-5df3f7d4ac63-kube-api-access-sqd5b\") pod \"isvc-canary-stop-predictor-cf655cf76-925g5\" (UID: \"86cff314-e9af-4027-91d9-5df3f7d4ac63\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-925g5"
Jul 28 10:36:40.983366 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:40.983284    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4fadb75f-28ee-409b-9c10-f4e16d1b7f10-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd\" (UID: \"4fadb75f-28ee-409b-9c10-f4e16d1b7f10\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd"
Jul 28 10:36:40.983366 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:40.983305    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4fadb75f-28ee-409b-9c10-f4e16d1b7f10-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd\" (UID: \"4fadb75f-28ee-409b-9c10-f4e16d1b7f10\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd"
Jul 28 10:36:40.983366 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:40.983359    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86cff314-e9af-4027-91d9-5df3f7d4ac63-proxy-tls\") pod \"isvc-canary-stop-predictor-cf655cf76-925g5\" (UID: \"86cff314-e9af-4027-91d9-5df3f7d4ac63\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-925g5"
Jul 28 10:36:40.983525 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:40.983413    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86cff314-e9af-4027-91d9-5df3f7d4ac63-kserve-provision-location\") pod \"isvc-canary-stop-predictor-cf655cf76-925g5\" (UID: \"86cff314-e9af-4027-91d9-5df3f7d4ac63\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-925g5"
Jul 28 10:36:40.983525 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:40.983442    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86cff314-e9af-4027-91d9-5df3f7d4ac63-cabundle-cert\") pod \"isvc-canary-stop-predictor-cf655cf76-925g5\" (UID: \"86cff314-e9af-4027-91d9-5df3f7d4ac63\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-925g5"
Jul 28 10:36:40.983525 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:40.983477    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2bl7\" (UniqueName: \"kubernetes.io/projected/4fadb75f-28ee-409b-9c10-f4e16d1b7f10-kube-api-access-l2bl7\") pod \"isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd\" (UID: \"4fadb75f-28ee-409b-9c10-f4e16d1b7f10\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd"
Jul 28 10:36:40.983817 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:40.983792    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4fadb75f-28ee-409b-9c10-f4e16d1b7f10-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd\" (UID: \"4fadb75f-28ee-409b-9c10-f4e16d1b7f10\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd"
Jul 28 10:36:40.983915 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:40.983885    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86cff314-e9af-4027-91d9-5df3f7d4ac63-kserve-provision-location\") pod \"isvc-canary-stop-predictor-cf655cf76-925g5\" (UID: \"86cff314-e9af-4027-91d9-5df3f7d4ac63\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-925g5"
Jul 28 10:36:40.984102 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:40.984077    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86cff314-e9af-4027-91d9-5df3f7d4ac63-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-cf655cf76-925g5\" (UID: \"86cff314-e9af-4027-91d9-5df3f7d4ac63\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-925g5"
Jul 28 10:36:40.984102 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:40.984092    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4fadb75f-28ee-409b-9c10-f4e16d1b7f10-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd\" (UID: \"4fadb75f-28ee-409b-9c10-f4e16d1b7f10\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd"
Jul 28 10:36:40.984277 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:40.984213    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4fadb75f-28ee-409b-9c10-f4e16d1b7f10-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd\" (UID: \"4fadb75f-28ee-409b-9c10-f4e16d1b7f10\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd"
Jul 28 10:36:40.984399 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:40.984380    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86cff314-e9af-4027-91d9-5df3f7d4ac63-cabundle-cert\") pod \"isvc-canary-stop-predictor-cf655cf76-925g5\" (UID: \"86cff314-e9af-4027-91d9-5df3f7d4ac63\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-925g5"
Jul 28 10:36:40.985864 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:40.985844    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4fadb75f-28ee-409b-9c10-f4e16d1b7f10-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd\" (UID: \"4fadb75f-28ee-409b-9c10-f4e16d1b7f10\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd"
Jul 28 10:36:40.986030 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:40.986015    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86cff314-e9af-4027-91d9-5df3f7d4ac63-proxy-tls\") pod \"isvc-canary-stop-predictor-cf655cf76-925g5\" (UID: \"86cff314-e9af-4027-91d9-5df3f7d4ac63\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-925g5"
Jul 28 10:36:40.991778 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:40.991758    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqd5b\" (UniqueName: \"kubernetes.io/projected/86cff314-e9af-4027-91d9-5df3f7d4ac63-kube-api-access-sqd5b\") pod \"isvc-canary-stop-predictor-cf655cf76-925g5\" (UID: \"86cff314-e9af-4027-91d9-5df3f7d4ac63\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-925g5"
Jul 28 10:36:40.991851 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:40.991764    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2bl7\" (UniqueName: \"kubernetes.io/projected/4fadb75f-28ee-409b-9c10-f4e16d1b7f10-kube-api-access-l2bl7\") pod \"isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd\" (UID: \"4fadb75f-28ee-409b-9c10-f4e16d1b7f10\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd"
Jul 28 10:36:41.072884 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:41.072841    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-925g5"
Jul 28 10:36:41.081010 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:41.080982    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd"
Jul 28 10:36:41.218424 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:41.218306    2583 generic.go:356] "Generic (PLEG): container finished" podID="92dda428-ac3b-4f25-8a19-143b08090311" containerID="0144d981414888a459b61088b1397c95c1b60099b8293017c4e8c4cfae39a1b2" exitCode=0
Jul 28 10:36:41.218424 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:41.218379    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-wglnd"
Jul 28 10:36:41.218424 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:41.218381    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-wglnd" event={"ID":"92dda428-ac3b-4f25-8a19-143b08090311","Type":"ContainerDied","Data":"0144d981414888a459b61088b1397c95c1b60099b8293017c4e8c4cfae39a1b2"}
Jul 28 10:36:41.218608 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:41.218453    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-wglnd" event={"ID":"92dda428-ac3b-4f25-8a19-143b08090311","Type":"ContainerDied","Data":"24f83e52ab0b3a23f2cda52991d8751f2365beffcbb0496818fde738b5a68dbe"}
Jul 28 10:36:41.218608 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:41.218497    2583 scope.go:117] "RemoveContainer" containerID="6e6f275bf70912d61bb2eba0f18cfbf4d5c2920861c686a8f256c43e6e9b71c9"
Jul 28 10:36:41.218770 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:41.218708    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-925g5"]
Jul 28 10:36:41.221400 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:36:41.221377    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86cff314_e9af_4027_91d9_5df3f7d4ac63.slice/crio-1d6a8bd2a3dea71c26009372946542bdc3c312620986313109df4c9ad8e0c28e WatchSource:0}: Error finding container 1d6a8bd2a3dea71c26009372946542bdc3c312620986313109df4c9ad8e0c28e: Status 404 returned error can't find the container with id 1d6a8bd2a3dea71c26009372946542bdc3c312620986313109df4c9ad8e0c28e
Jul 28 10:36:41.228025 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:41.228010    2583 scope.go:117] "RemoveContainer" containerID="0144d981414888a459b61088b1397c95c1b60099b8293017c4e8c4cfae39a1b2"
Jul 28 10:36:41.238552 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:41.238529    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd"]
Jul 28 10:36:41.239454 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:36:41.239414    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fadb75f_28ee_409b_9c10_f4e16d1b7f10.slice/crio-a066012df6901947a3f47519594cc4bc1b176a0075e2bc40703b741400ab59fd WatchSource:0}: Error finding container a066012df6901947a3f47519594cc4bc1b176a0075e2bc40703b741400ab59fd: Status 404 returned error can't find the container with id a066012df6901947a3f47519594cc4bc1b176a0075e2bc40703b741400ab59fd
Jul 28 10:36:41.249106 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:41.248975    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-wglnd"]
Jul 28 10:36:41.249106 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:41.249026    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-wglnd"]
Jul 28 10:36:41.261548 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:41.261526    2583 scope.go:117] "RemoveContainer" containerID="3959245e2163460d16993bb9690dfdfd04b647f34e720202a2129ee5dca401a0"
Jul 28 10:36:41.277998 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:41.277979    2583 scope.go:117] "RemoveContainer" containerID="6e6f275bf70912d61bb2eba0f18cfbf4d5c2920861c686a8f256c43e6e9b71c9"
Jul 28 10:36:41.278321 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:36:41.278296    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e6f275bf70912d61bb2eba0f18cfbf4d5c2920861c686a8f256c43e6e9b71c9\": container with ID starting with 6e6f275bf70912d61bb2eba0f18cfbf4d5c2920861c686a8f256c43e6e9b71c9 not found: ID does not exist" containerID="6e6f275bf70912d61bb2eba0f18cfbf4d5c2920861c686a8f256c43e6e9b71c9"
Jul 28 10:36:41.278398 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:41.278333    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e6f275bf70912d61bb2eba0f18cfbf4d5c2920861c686a8f256c43e6e9b71c9"} err="failed to get container status \"6e6f275bf70912d61bb2eba0f18cfbf4d5c2920861c686a8f256c43e6e9b71c9\": rpc error: code = NotFound desc = could not find container \"6e6f275bf70912d61bb2eba0f18cfbf4d5c2920861c686a8f256c43e6e9b71c9\": container with ID starting with 6e6f275bf70912d61bb2eba0f18cfbf4d5c2920861c686a8f256c43e6e9b71c9 not found: ID does not exist"
Jul 28 10:36:41.278398 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:41.278360    2583 scope.go:117] "RemoveContainer" containerID="0144d981414888a459b61088b1397c95c1b60099b8293017c4e8c4cfae39a1b2"
Jul 28 10:36:41.278640 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:36:41.278622    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0144d981414888a459b61088b1397c95c1b60099b8293017c4e8c4cfae39a1b2\": container with ID starting with 0144d981414888a459b61088b1397c95c1b60099b8293017c4e8c4cfae39a1b2 not found: ID does not exist" containerID="0144d981414888a459b61088b1397c95c1b60099b8293017c4e8c4cfae39a1b2"
Jul 28 10:36:41.278706 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:41.278649    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0144d981414888a459b61088b1397c95c1b60099b8293017c4e8c4cfae39a1b2"} err="failed to get container status \"0144d981414888a459b61088b1397c95c1b60099b8293017c4e8c4cfae39a1b2\": rpc error: code = NotFound desc = could not find container \"0144d981414888a459b61088b1397c95c1b60099b8293017c4e8c4cfae39a1b2\": container with ID starting with 0144d981414888a459b61088b1397c95c1b60099b8293017c4e8c4cfae39a1b2 not found: ID does not exist"
Jul 28 10:36:41.278706 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:41.278674    2583 scope.go:117] "RemoveContainer" containerID="3959245e2163460d16993bb9690dfdfd04b647f34e720202a2129ee5dca401a0"
Jul 28 10:36:41.278900 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:36:41.278881    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3959245e2163460d16993bb9690dfdfd04b647f34e720202a2129ee5dca401a0\": container with ID starting with 3959245e2163460d16993bb9690dfdfd04b647f34e720202a2129ee5dca401a0 not found: ID does not exist" containerID="3959245e2163460d16993bb9690dfdfd04b647f34e720202a2129ee5dca401a0"
Jul 28 10:36:41.278942 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:41.278907    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3959245e2163460d16993bb9690dfdfd04b647f34e720202a2129ee5dca401a0"} err="failed to get container status \"3959245e2163460d16993bb9690dfdfd04b647f34e720202a2129ee5dca401a0\": rpc error: code = NotFound desc = could not find container \"3959245e2163460d16993bb9690dfdfd04b647f34e720202a2129ee5dca401a0\": container with ID starting with 3959245e2163460d16993bb9690dfdfd04b647f34e720202a2129ee5dca401a0 not found: ID does not exist"
Jul 28 10:36:41.441257 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:41.441220    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92dda428-ac3b-4f25-8a19-143b08090311" path="/var/lib/kubelet/pods/92dda428-ac3b-4f25-8a19-143b08090311/volumes"
Jul 28 10:36:42.223828 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:42.223787    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd" event={"ID":"4fadb75f-28ee-409b-9c10-f4e16d1b7f10","Type":"ContainerStarted","Data":"f2c993ef30f70467ba18446fe8a0ec2da3a5b136a7a235cc7ac1de34cc3651a5"}
Jul 28 10:36:42.224283 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:42.223862    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd" event={"ID":"4fadb75f-28ee-409b-9c10-f4e16d1b7f10","Type":"ContainerStarted","Data":"a066012df6901947a3f47519594cc4bc1b176a0075e2bc40703b741400ab59fd"}
Jul 28 10:36:42.225318 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:42.225292    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-925g5" event={"ID":"86cff314-e9af-4027-91d9-5df3f7d4ac63","Type":"ContainerStarted","Data":"724ab30a9c3962bcc5ab93f2ee762cf7132b9f76d368c43a4b9561eb0fc8593a"}
Jul 28 10:36:42.225373 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:42.225351    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-925g5" event={"ID":"86cff314-e9af-4027-91d9-5df3f7d4ac63","Type":"ContainerStarted","Data":"1d6a8bd2a3dea71c26009372946542bdc3c312620986313109df4c9ad8e0c28e"}
Jul 28 10:36:45.240049 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:45.239961    2583 generic.go:356] "Generic (PLEG): container finished" podID="4fadb75f-28ee-409b-9c10-f4e16d1b7f10" containerID="f2c993ef30f70467ba18446fe8a0ec2da3a5b136a7a235cc7ac1de34cc3651a5" exitCode=0
Jul 28 10:36:45.240049 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:45.240035    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd" event={"ID":"4fadb75f-28ee-409b-9c10-f4e16d1b7f10","Type":"ContainerDied","Data":"f2c993ef30f70467ba18446fe8a0ec2da3a5b136a7a235cc7ac1de34cc3651a5"}
Jul 28 10:36:45.241670 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:45.241645    2583 generic.go:356] "Generic (PLEG): container finished" podID="86cff314-e9af-4027-91d9-5df3f7d4ac63" containerID="724ab30a9c3962bcc5ab93f2ee762cf7132b9f76d368c43a4b9561eb0fc8593a" exitCode=0
Jul 28 10:36:45.241771 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:45.241720    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-925g5" event={"ID":"86cff314-e9af-4027-91d9-5df3f7d4ac63","Type":"ContainerDied","Data":"724ab30a9c3962bcc5ab93f2ee762cf7132b9f76d368c43a4b9561eb0fc8593a"}
Jul 28 10:36:46.248342 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:46.248302    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd" event={"ID":"4fadb75f-28ee-409b-9c10-f4e16d1b7f10","Type":"ContainerStarted","Data":"4bc761f1a72ef5e5aca05588a86625d1709cf3927c1467d4898b84da630c8e19"}
Jul 28 10:36:46.248783 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:46.248368    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd" event={"ID":"4fadb75f-28ee-409b-9c10-f4e16d1b7f10","Type":"ContainerStarted","Data":"e110f671316ee3081bdb36135c748c49eddfa5a04ea4190b1c89fc9da9519e18"}
Jul 28 10:36:46.248783 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:46.248644    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd"
Jul 28 10:36:46.248918 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:46.248822    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd"
Jul 28 10:36:46.250137 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:46.250109    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd" podUID="4fadb75f-28ee-409b-9c10-f4e16d1b7f10" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 10:36:46.250619 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:46.250585    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-925g5" event={"ID":"86cff314-e9af-4027-91d9-5df3f7d4ac63","Type":"ContainerStarted","Data":"1586adab558b3d6fe06c36be006b5530d8cd54a2974ccfe0f88099f697f38b3f"}
Jul 28 10:36:46.250690 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:46.250647    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-925g5" event={"ID":"86cff314-e9af-4027-91d9-5df3f7d4ac63","Type":"ContainerStarted","Data":"668f5924c35d1748db4c75508b57720d7212bc4328cc8a259ab53b572a41515b"}
Jul 28 10:36:46.250989 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:46.250971    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-925g5"
Jul 28 10:36:46.251046 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:46.251012    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-925g5"
Jul 28 10:36:46.251912 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:46.251890    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-925g5" podUID="86cff314-e9af-4027-91d9-5df3f7d4ac63" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 10:36:46.272354 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:46.272311    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd" podStartSLOduration=2.6437829170000002 podStartE2EDuration="6.272297641s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.628514724s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:36:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:36:46.269395033 +0000 UTC m=+1745.369901600" watchObservedRunningTime="2026-07-28 10:36:46.272297641 +0000 UTC m=+1745.372804195"
Jul 28 10:36:46.295127 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:46.295084    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-925g5" podStartSLOduration=2.624299786 podStartE2EDuration="6.295070437s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.670770651s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:36:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:36:46.2937767 +0000 UTC m=+1745.394283246" watchObservedRunningTime="2026-07-28 10:36:46.295070437 +0000 UTC m=+1745.395576992"
Jul 28 10:36:47.255621 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:47.255584    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-925g5" podUID="86cff314-e9af-4027-91d9-5df3f7d4ac63" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 10:36:47.256053 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:47.255737    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd" podUID="4fadb75f-28ee-409b-9c10-f4e16d1b7f10" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 10:36:52.260682 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:52.260649    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd"
Jul 28 10:36:52.261322 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:52.261296    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-925g5"
Jul 28 10:36:52.261865 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:52.261835    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd" podUID="4fadb75f-28ee-409b-9c10-f4e16d1b7f10" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 10:36:52.262083 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:36:52.262061    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-925g5" podUID="86cff314-e9af-4027-91d9-5df3f7d4ac63" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 10:37:02.262633 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:37:02.262593    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd" podUID="4fadb75f-28ee-409b-9c10-f4e16d1b7f10" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 10:37:02.263029 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:37:02.262592    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-925g5" podUID="86cff314-e9af-4027-91d9-5df3f7d4ac63" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 10:37:12.262080 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:37:12.262039    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd" podUID="4fadb75f-28ee-409b-9c10-f4e16d1b7f10" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 10:37:12.262647 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:37:12.262045    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-925g5" podUID="86cff314-e9af-4027-91d9-5df3f7d4ac63" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 10:37:22.262031 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:37:22.261989    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-925g5" podUID="86cff314-e9af-4027-91d9-5df3f7d4ac63" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 10:37:22.262435 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:37:22.261989    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd" podUID="4fadb75f-28ee-409b-9c10-f4e16d1b7f10" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 10:37:32.261682 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:37:32.261641    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd" podUID="4fadb75f-28ee-409b-9c10-f4e16d1b7f10" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 10:37:32.262071 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:37:32.261773    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-925g5" podUID="86cff314-e9af-4027-91d9-5df3f7d4ac63" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 10:37:42.262236 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:37:42.262191    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-925g5" podUID="86cff314-e9af-4027-91d9-5df3f7d4ac63" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 10:37:42.262671 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:37:42.262200    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd" podUID="4fadb75f-28ee-409b-9c10-f4e16d1b7f10" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 10:37:52.262738 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:37:52.262698    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd" podUID="4fadb75f-28ee-409b-9c10-f4e16d1b7f10" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 10:37:52.263198 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:37:52.262694    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-925g5" podUID="86cff314-e9af-4027-91d9-5df3f7d4ac63" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 10:37:54.433840 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:37:54.433785    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd" podUID="4fadb75f-28ee-409b-9c10-f4e16d1b7f10" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 10:38:02.263195 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:02.263160    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-925g5"
Jul 28 10:38:04.435198 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:04.435143    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd"
Jul 28 10:38:10.780772 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:10.780737    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-925g5"]
Jul 28 10:38:10.781253 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:10.781035    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-925g5" podUID="86cff314-e9af-4027-91d9-5df3f7d4ac63" containerName="kserve-container" containerID="cri-o://668f5924c35d1748db4c75508b57720d7212bc4328cc8a259ab53b572a41515b" gracePeriod=30
Jul 28 10:38:10.781324 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:10.781259    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-925g5" podUID="86cff314-e9af-4027-91d9-5df3f7d4ac63" containerName="kube-rbac-proxy" containerID="cri-o://1586adab558b3d6fe06c36be006b5530d8cd54a2974ccfe0f88099f697f38b3f" gracePeriod=30
Jul 28 10:38:10.800770 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:10.800743    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:38:10.810842 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:10.810810    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd"]
Jul 28 10:38:10.811082 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:10.811058    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd" podUID="4fadb75f-28ee-409b-9c10-f4e16d1b7f10" containerName="kserve-container" containerID="cri-o://e110f671316ee3081bdb36135c748c49eddfa5a04ea4190b1c89fc9da9519e18" gracePeriod=30
Jul 28 10:38:10.811137 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:10.811079    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd" podUID="4fadb75f-28ee-409b-9c10-f4e16d1b7f10" containerName="kube-rbac-proxy" containerID="cri-o://4bc761f1a72ef5e5aca05588a86625d1709cf3927c1467d4898b84da630c8e19" gracePeriod=30
Jul 28 10:38:10.898023 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:10.897999    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:38:11.630045 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:11.630014    2583 generic.go:356] "Generic (PLEG): container finished" podID="4fadb75f-28ee-409b-9c10-f4e16d1b7f10" containerID="4bc761f1a72ef5e5aca05588a86625d1709cf3927c1467d4898b84da630c8e19" exitCode=2
Jul 28 10:38:11.630248 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:11.630092    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd" event={"ID":"4fadb75f-28ee-409b-9c10-f4e16d1b7f10","Type":"ContainerDied","Data":"4bc761f1a72ef5e5aca05588a86625d1709cf3927c1467d4898b84da630c8e19"}
Jul 28 10:38:11.632093 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:11.632071    2583 generic.go:356] "Generic (PLEG): container finished" podID="86cff314-e9af-4027-91d9-5df3f7d4ac63" containerID="1586adab558b3d6fe06c36be006b5530d8cd54a2974ccfe0f88099f697f38b3f" exitCode=2
Jul 28 10:38:11.632231 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:11.632097    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-925g5" event={"ID":"86cff314-e9af-4027-91d9-5df3f7d4ac63","Type":"ContainerDied","Data":"1586adab558b3d6fe06c36be006b5530d8cd54a2974ccfe0f88099f697f38b3f"}
Jul 28 10:38:12.256593 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:12.256546    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-925g5" podUID="86cff314-e9af-4027-91d9-5df3f7d4ac63" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused"
Jul 28 10:38:12.256975 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:12.256553    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd" podUID="4fadb75f-28ee-409b-9c10-f4e16d1b7f10" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.40:8643/healthz\": dial tcp 10.134.0.40:8643: connect: connection refused"
Jul 28 10:38:12.262090 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:12.262063    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-925g5" podUID="86cff314-e9af-4027-91d9-5df3f7d4ac63" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 10:38:14.433842 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:14.433804    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd" podUID="4fadb75f-28ee-409b-9c10-f4e16d1b7f10" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 10:38:15.624151 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:15.624122    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-925g5"
Jul 28 10:38:15.656089 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:15.656054    2583 generic.go:356] "Generic (PLEG): container finished" podID="86cff314-e9af-4027-91d9-5df3f7d4ac63" containerID="668f5924c35d1748db4c75508b57720d7212bc4328cc8a259ab53b572a41515b" exitCode=0
Jul 28 10:38:15.656208 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:15.656098    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-925g5" event={"ID":"86cff314-e9af-4027-91d9-5df3f7d4ac63","Type":"ContainerDied","Data":"668f5924c35d1748db4c75508b57720d7212bc4328cc8a259ab53b572a41515b"}
Jul 28 10:38:15.656208 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:15.656124    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-925g5"
Jul 28 10:38:15.656208 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:15.656162    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-925g5" event={"ID":"86cff314-e9af-4027-91d9-5df3f7d4ac63","Type":"ContainerDied","Data":"1d6a8bd2a3dea71c26009372946542bdc3c312620986313109df4c9ad8e0c28e"}
Jul 28 10:38:15.656343 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:15.656208    2583 scope.go:117] "RemoveContainer" containerID="1586adab558b3d6fe06c36be006b5530d8cd54a2974ccfe0f88099f697f38b3f"
Jul 28 10:38:15.721365 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:15.721346    2583 scope.go:117] "RemoveContainer" containerID="668f5924c35d1748db4c75508b57720d7212bc4328cc8a259ab53b572a41515b"
Jul 28 10:38:15.728720 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:15.728704    2583 scope.go:117] "RemoveContainer" containerID="724ab30a9c3962bcc5ab93f2ee762cf7132b9f76d368c43a4b9561eb0fc8593a"
Jul 28 10:38:15.737381 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:15.737355    2583 scope.go:117] "RemoveContainer" containerID="1586adab558b3d6fe06c36be006b5530d8cd54a2974ccfe0f88099f697f38b3f"
Jul 28 10:38:15.737654 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:38:15.737628    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1586adab558b3d6fe06c36be006b5530d8cd54a2974ccfe0f88099f697f38b3f\": container with ID starting with 1586adab558b3d6fe06c36be006b5530d8cd54a2974ccfe0f88099f697f38b3f not found: ID does not exist" containerID="1586adab558b3d6fe06c36be006b5530d8cd54a2974ccfe0f88099f697f38b3f"
Jul 28 10:38:15.737721 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:15.737662    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1586adab558b3d6fe06c36be006b5530d8cd54a2974ccfe0f88099f697f38b3f"} err="failed to get container status \"1586adab558b3d6fe06c36be006b5530d8cd54a2974ccfe0f88099f697f38b3f\": rpc error: code = NotFound desc = could not find container \"1586adab558b3d6fe06c36be006b5530d8cd54a2974ccfe0f88099f697f38b3f\": container with ID starting with 1586adab558b3d6fe06c36be006b5530d8cd54a2974ccfe0f88099f697f38b3f not found: ID does not exist"
Jul 28 10:38:15.737721 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:15.737683    2583 scope.go:117] "RemoveContainer" containerID="668f5924c35d1748db4c75508b57720d7212bc4328cc8a259ab53b572a41515b"
Jul 28 10:38:15.737889 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:38:15.737874    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"668f5924c35d1748db4c75508b57720d7212bc4328cc8a259ab53b572a41515b\": container with ID starting with 668f5924c35d1748db4c75508b57720d7212bc4328cc8a259ab53b572a41515b not found: ID does not exist" containerID="668f5924c35d1748db4c75508b57720d7212bc4328cc8a259ab53b572a41515b"
Jul 28 10:38:15.737935 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:15.737893    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"668f5924c35d1748db4c75508b57720d7212bc4328cc8a259ab53b572a41515b"} err="failed to get container status \"668f5924c35d1748db4c75508b57720d7212bc4328cc8a259ab53b572a41515b\": rpc error: code = NotFound desc = could not find container \"668f5924c35d1748db4c75508b57720d7212bc4328cc8a259ab53b572a41515b\": container with ID starting with 668f5924c35d1748db4c75508b57720d7212bc4328cc8a259ab53b572a41515b not found: ID does not exist"
Jul 28 10:38:15.737935 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:15.737906    2583 scope.go:117] "RemoveContainer" containerID="724ab30a9c3962bcc5ab93f2ee762cf7132b9f76d368c43a4b9561eb0fc8593a"
Jul 28 10:38:15.738146 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:38:15.738131    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"724ab30a9c3962bcc5ab93f2ee762cf7132b9f76d368c43a4b9561eb0fc8593a\": container with ID starting with 724ab30a9c3962bcc5ab93f2ee762cf7132b9f76d368c43a4b9561eb0fc8593a not found: ID does not exist" containerID="724ab30a9c3962bcc5ab93f2ee762cf7132b9f76d368c43a4b9561eb0fc8593a"
Jul 28 10:38:15.738224 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:15.738149    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"724ab30a9c3962bcc5ab93f2ee762cf7132b9f76d368c43a4b9561eb0fc8593a"} err="failed to get container status \"724ab30a9c3962bcc5ab93f2ee762cf7132b9f76d368c43a4b9561eb0fc8593a\": rpc error: code = NotFound desc = could not find container \"724ab30a9c3962bcc5ab93f2ee762cf7132b9f76d368c43a4b9561eb0fc8593a\": container with ID starting with 724ab30a9c3962bcc5ab93f2ee762cf7132b9f76d368c43a4b9561eb0fc8593a not found: ID does not exist"
Jul 28 10:38:15.798664 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:15.798633    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86cff314-e9af-4027-91d9-5df3f7d4ac63-cabundle-cert\") pod \"86cff314-e9af-4027-91d9-5df3f7d4ac63\" (UID: \"86cff314-e9af-4027-91d9-5df3f7d4ac63\") "
Jul 28 10:38:15.798795 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:15.798685    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86cff314-e9af-4027-91d9-5df3f7d4ac63-kserve-provision-location\") pod \"86cff314-e9af-4027-91d9-5df3f7d4ac63\" (UID: \"86cff314-e9af-4027-91d9-5df3f7d4ac63\") "
Jul 28 10:38:15.798795 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:15.798724    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86cff314-e9af-4027-91d9-5df3f7d4ac63-proxy-tls\") pod \"86cff314-e9af-4027-91d9-5df3f7d4ac63\" (UID: \"86cff314-e9af-4027-91d9-5df3f7d4ac63\") "
Jul 28 10:38:15.798919 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:15.798810    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sqd5b\" (UniqueName: \"kubernetes.io/projected/86cff314-e9af-4027-91d9-5df3f7d4ac63-kube-api-access-sqd5b\") pod \"86cff314-e9af-4027-91d9-5df3f7d4ac63\" (UID: \"86cff314-e9af-4027-91d9-5df3f7d4ac63\") "
Jul 28 10:38:15.798919 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:15.798836    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86cff314-e9af-4027-91d9-5df3f7d4ac63-kube-rbac-proxy-sar-config\") pod \"86cff314-e9af-4027-91d9-5df3f7d4ac63\" (UID: \"86cff314-e9af-4027-91d9-5df3f7d4ac63\") "
Jul 28 10:38:15.799025 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:15.798996    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86cff314-e9af-4027-91d9-5df3f7d4ac63-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "86cff314-e9af-4027-91d9-5df3f7d4ac63" (UID: "86cff314-e9af-4027-91d9-5df3f7d4ac63"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:38:15.799126 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:15.799101    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86cff314-e9af-4027-91d9-5df3f7d4ac63-cabundle-cert\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:38:15.799326 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:15.799279    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86cff314-e9af-4027-91d9-5df3f7d4ac63-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "86cff314-e9af-4027-91d9-5df3f7d4ac63" (UID: "86cff314-e9af-4027-91d9-5df3f7d4ac63"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:38:15.799427 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:15.799340    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86cff314-e9af-4027-91d9-5df3f7d4ac63-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "86cff314-e9af-4027-91d9-5df3f7d4ac63" (UID: "86cff314-e9af-4027-91d9-5df3f7d4ac63"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:38:15.801906 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:15.801872    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86cff314-e9af-4027-91d9-5df3f7d4ac63-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "86cff314-e9af-4027-91d9-5df3f7d4ac63" (UID: "86cff314-e9af-4027-91d9-5df3f7d4ac63"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:38:15.802387 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:15.802365    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86cff314-e9af-4027-91d9-5df3f7d4ac63-kube-api-access-sqd5b" (OuterVolumeSpecName: "kube-api-access-sqd5b") pod "86cff314-e9af-4027-91d9-5df3f7d4ac63" (UID: "86cff314-e9af-4027-91d9-5df3f7d4ac63"). InnerVolumeSpecName "kube-api-access-sqd5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:38:15.849292 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:15.849271    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd"
Jul 28 10:38:15.899620 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:15.899593    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86cff314-e9af-4027-91d9-5df3f7d4ac63-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:38:15.899620 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:15.899619    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86cff314-e9af-4027-91d9-5df3f7d4ac63-kserve-provision-location\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:38:15.899789 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:15.899628    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86cff314-e9af-4027-91d9-5df3f7d4ac63-proxy-tls\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:38:15.899789 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:15.899637    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqd5b\" (UniqueName: \"kubernetes.io/projected/86cff314-e9af-4027-91d9-5df3f7d4ac63-kube-api-access-sqd5b\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:38:15.977571 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:15.977514    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-925g5"]
Jul 28 10:38:15.983145 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:15.983124    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-925g5"]
Jul 28 10:38:15.999945 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:15.999924    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l2bl7\" (UniqueName: \"kubernetes.io/projected/4fadb75f-28ee-409b-9c10-f4e16d1b7f10-kube-api-access-l2bl7\") pod \"4fadb75f-28ee-409b-9c10-f4e16d1b7f10\" (UID: \"4fadb75f-28ee-409b-9c10-f4e16d1b7f10\") "
Jul 28 10:38:16.000034 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:15.999974    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4fadb75f-28ee-409b-9c10-f4e16d1b7f10-proxy-tls\") pod \"4fadb75f-28ee-409b-9c10-f4e16d1b7f10\" (UID: \"4fadb75f-28ee-409b-9c10-f4e16d1b7f10\") "
Jul 28 10:38:16.000034 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:16.000015    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4fadb75f-28ee-409b-9c10-f4e16d1b7f10-kserve-provision-location\") pod \"4fadb75f-28ee-409b-9c10-f4e16d1b7f10\" (UID: \"4fadb75f-28ee-409b-9c10-f4e16d1b7f10\") "
Jul 28 10:38:16.000147 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:16.000036    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4fadb75f-28ee-409b-9c10-f4e16d1b7f10-cabundle-cert\") pod \"4fadb75f-28ee-409b-9c10-f4e16d1b7f10\" (UID: \"4fadb75f-28ee-409b-9c10-f4e16d1b7f10\") "
Jul 28 10:38:16.000147 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:16.000064    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4fadb75f-28ee-409b-9c10-f4e16d1b7f10-kube-rbac-proxy-sar-config\") pod \"4fadb75f-28ee-409b-9c10-f4e16d1b7f10\" (UID: \"4fadb75f-28ee-409b-9c10-f4e16d1b7f10\") "
Jul 28 10:38:16.000413 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:16.000385    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fadb75f-28ee-409b-9c10-f4e16d1b7f10-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4fadb75f-28ee-409b-9c10-f4e16d1b7f10" (UID: "4fadb75f-28ee-409b-9c10-f4e16d1b7f10"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:38:16.000492 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:16.000387    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fadb75f-28ee-409b-9c10-f4e16d1b7f10-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4fadb75f-28ee-409b-9c10-f4e16d1b7f10" (UID: "4fadb75f-28ee-409b-9c10-f4e16d1b7f10"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:38:16.000492 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:16.000440    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fadb75f-28ee-409b-9c10-f4e16d1b7f10-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4fadb75f-28ee-409b-9c10-f4e16d1b7f10" (UID: "4fadb75f-28ee-409b-9c10-f4e16d1b7f10"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:38:16.002043 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:16.002023    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fadb75f-28ee-409b-9c10-f4e16d1b7f10-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4fadb75f-28ee-409b-9c10-f4e16d1b7f10" (UID: "4fadb75f-28ee-409b-9c10-f4e16d1b7f10"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:38:16.002097 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:16.002047    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fadb75f-28ee-409b-9c10-f4e16d1b7f10-kube-api-access-l2bl7" (OuterVolumeSpecName: "kube-api-access-l2bl7") pod "4fadb75f-28ee-409b-9c10-f4e16d1b7f10" (UID: "4fadb75f-28ee-409b-9c10-f4e16d1b7f10"). InnerVolumeSpecName "kube-api-access-l2bl7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:38:16.101139 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:16.101119    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2bl7\" (UniqueName: \"kubernetes.io/projected/4fadb75f-28ee-409b-9c10-f4e16d1b7f10-kube-api-access-l2bl7\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:38:16.101139 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:16.101140    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4fadb75f-28ee-409b-9c10-f4e16d1b7f10-proxy-tls\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:38:16.101275 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:16.101150    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4fadb75f-28ee-409b-9c10-f4e16d1b7f10-kserve-provision-location\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:38:16.101275 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:16.101159    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4fadb75f-28ee-409b-9c10-f4e16d1b7f10-cabundle-cert\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:38:16.101275 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:16.101167    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4fadb75f-28ee-409b-9c10-f4e16d1b7f10-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:38:16.662249 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:16.662212    2583 generic.go:356] "Generic (PLEG): container finished" podID="4fadb75f-28ee-409b-9c10-f4e16d1b7f10" containerID="e110f671316ee3081bdb36135c748c49eddfa5a04ea4190b1c89fc9da9519e18" exitCode=0
Jul 28 10:38:16.662655 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:16.662279    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd"
Jul 28 10:38:16.662655 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:16.662294    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd" event={"ID":"4fadb75f-28ee-409b-9c10-f4e16d1b7f10","Type":"ContainerDied","Data":"e110f671316ee3081bdb36135c748c49eddfa5a04ea4190b1c89fc9da9519e18"}
Jul 28 10:38:16.662655 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:16.662378    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd" event={"ID":"4fadb75f-28ee-409b-9c10-f4e16d1b7f10","Type":"ContainerDied","Data":"a066012df6901947a3f47519594cc4bc1b176a0075e2bc40703b741400ab59fd"}
Jul 28 10:38:16.662655 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:16.662416    2583 scope.go:117] "RemoveContainer" containerID="4bc761f1a72ef5e5aca05588a86625d1709cf3927c1467d4898b84da630c8e19"
Jul 28 10:38:16.671357 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:16.671341    2583 scope.go:117] "RemoveContainer" containerID="e110f671316ee3081bdb36135c748c49eddfa5a04ea4190b1c89fc9da9519e18"
Jul 28 10:38:16.678505 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:16.678487    2583 scope.go:117] "RemoveContainer" containerID="f2c993ef30f70467ba18446fe8a0ec2da3a5b136a7a235cc7ac1de34cc3651a5"
Jul 28 10:38:16.684515 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:16.684494    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd"]
Jul 28 10:38:16.685951 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:16.685932    2583 scope.go:117] "RemoveContainer" containerID="4bc761f1a72ef5e5aca05588a86625d1709cf3927c1467d4898b84da630c8e19"
Jul 28 10:38:16.686214 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:38:16.686195    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bc761f1a72ef5e5aca05588a86625d1709cf3927c1467d4898b84da630c8e19\": container with ID starting with 4bc761f1a72ef5e5aca05588a86625d1709cf3927c1467d4898b84da630c8e19 not found: ID does not exist" containerID="4bc761f1a72ef5e5aca05588a86625d1709cf3927c1467d4898b84da630c8e19"
Jul 28 10:38:16.686318 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:16.686227    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bc761f1a72ef5e5aca05588a86625d1709cf3927c1467d4898b84da630c8e19"} err="failed to get container status \"4bc761f1a72ef5e5aca05588a86625d1709cf3927c1467d4898b84da630c8e19\": rpc error: code = NotFound desc = could not find container \"4bc761f1a72ef5e5aca05588a86625d1709cf3927c1467d4898b84da630c8e19\": container with ID starting with 4bc761f1a72ef5e5aca05588a86625d1709cf3927c1467d4898b84da630c8e19 not found: ID does not exist"
Jul 28 10:38:16.686318 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:16.686251    2583 scope.go:117] "RemoveContainer" containerID="e110f671316ee3081bdb36135c748c49eddfa5a04ea4190b1c89fc9da9519e18"
Jul 28 10:38:16.686512 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:38:16.686495    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e110f671316ee3081bdb36135c748c49eddfa5a04ea4190b1c89fc9da9519e18\": container with ID starting with e110f671316ee3081bdb36135c748c49eddfa5a04ea4190b1c89fc9da9519e18 not found: ID does not exist" containerID="e110f671316ee3081bdb36135c748c49eddfa5a04ea4190b1c89fc9da9519e18"
Jul 28 10:38:16.686558 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:16.686519    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e110f671316ee3081bdb36135c748c49eddfa5a04ea4190b1c89fc9da9519e18"} err="failed to get container status \"e110f671316ee3081bdb36135c748c49eddfa5a04ea4190b1c89fc9da9519e18\": rpc error: code = NotFound desc = could not find container \"e110f671316ee3081bdb36135c748c49eddfa5a04ea4190b1c89fc9da9519e18\": container with ID starting with e110f671316ee3081bdb36135c748c49eddfa5a04ea4190b1c89fc9da9519e18 not found: ID does not exist"
Jul 28 10:38:16.686558 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:16.686536    2583 scope.go:117] "RemoveContainer" containerID="f2c993ef30f70467ba18446fe8a0ec2da3a5b136a7a235cc7ac1de34cc3651a5"
Jul 28 10:38:16.686783 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:38:16.686765    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2c993ef30f70467ba18446fe8a0ec2da3a5b136a7a235cc7ac1de34cc3651a5\": container with ID starting with f2c993ef30f70467ba18446fe8a0ec2da3a5b136a7a235cc7ac1de34cc3651a5 not found: ID does not exist" containerID="f2c993ef30f70467ba18446fe8a0ec2da3a5b136a7a235cc7ac1de34cc3651a5"
Jul 28 10:38:16.686845 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:16.686790    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2c993ef30f70467ba18446fe8a0ec2da3a5b136a7a235cc7ac1de34cc3651a5"} err="failed to get container status \"f2c993ef30f70467ba18446fe8a0ec2da3a5b136a7a235cc7ac1de34cc3651a5\": rpc error: code = NotFound desc = could not find container \"f2c993ef30f70467ba18446fe8a0ec2da3a5b136a7a235cc7ac1de34cc3651a5\": container with ID starting with f2c993ef30f70467ba18446fe8a0ec2da3a5b136a7a235cc7ac1de34cc3651a5 not found: ID does not exist"
Jul 28 10:38:16.689268 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:16.689250    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-4jpsd"]
Jul 28 10:38:17.438567 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:17.438534    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4fadb75f-28ee-409b-9c10-f4e16d1b7f10" path="/var/lib/kubelet/pods/4fadb75f-28ee-409b-9c10-f4e16d1b7f10/volumes"
Jul 28 10:38:17.439036 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:17.439023    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86cff314-e9af-4027-91d9-5df3f7d4ac63" path="/var/lib/kubelet/pods/86cff314-e9af-4027-91d9-5df3f7d4ac63/volumes"
Jul 28 10:38:21.022203 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:21.022147    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:38:21.093173 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:21.093140    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-1e74d8-predictor-6dfc9b49f6-2q5xs"]
Jul 28 10:38:21.093546 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:21.093531    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fadb75f-28ee-409b-9c10-f4e16d1b7f10" containerName="kube-rbac-proxy"
Jul 28 10:38:21.093594 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:21.093548    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fadb75f-28ee-409b-9c10-f4e16d1b7f10" containerName="kube-rbac-proxy"
Jul 28 10:38:21.093594 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:21.093563    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fadb75f-28ee-409b-9c10-f4e16d1b7f10" containerName="kserve-container"
Jul 28 10:38:21.093594 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:21.093569    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fadb75f-28ee-409b-9c10-f4e16d1b7f10" containerName="kserve-container"
Jul 28 10:38:21.093594 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:21.093579    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86cff314-e9af-4027-91d9-5df3f7d4ac63" containerName="storage-initializer"
Jul 28 10:38:21.093594 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:21.093585    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="86cff314-e9af-4027-91d9-5df3f7d4ac63" containerName="storage-initializer"
Jul 28 10:38:21.093755 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:21.093608    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86cff314-e9af-4027-91d9-5df3f7d4ac63" containerName="kserve-container"
Jul 28 10:38:21.093755 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:21.093617    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="86cff314-e9af-4027-91d9-5df3f7d4ac63" containerName="kserve-container"
Jul 28 10:38:21.093755 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:21.093629    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86cff314-e9af-4027-91d9-5df3f7d4ac63" containerName="kube-rbac-proxy"
Jul 28 10:38:21.093755 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:21.093635    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="86cff314-e9af-4027-91d9-5df3f7d4ac63" containerName="kube-rbac-proxy"
Jul 28 10:38:21.093755 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:21.093648    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fadb75f-28ee-409b-9c10-f4e16d1b7f10" containerName="storage-initializer"
Jul 28 10:38:21.093755 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:21.093653    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fadb75f-28ee-409b-9c10-f4e16d1b7f10" containerName="storage-initializer"
Jul 28 10:38:21.093755 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:21.093706    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fadb75f-28ee-409b-9c10-f4e16d1b7f10" containerName="kube-rbac-proxy"
Jul 28 10:38:21.093755 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:21.093714    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fadb75f-28ee-409b-9c10-f4e16d1b7f10" containerName="kserve-container"
Jul 28 10:38:21.093755 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:21.093723    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="86cff314-e9af-4027-91d9-5df3f7d4ac63" containerName="kube-rbac-proxy"
Jul 28 10:38:21.093755 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:21.093731    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="86cff314-e9af-4027-91d9-5df3f7d4ac63" containerName="kserve-container"
Jul 28 10:38:21.098592 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:21.098572    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-1e74d8-predictor-6dfc9b49f6-2q5xs"
Jul 28 10:38:21.103690 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:21.103669    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 10:38:21.103948 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:21.103914    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-1e74d8-predictor-serving-cert\""
Jul 28 10:38:21.104068 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:21.103950    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 10:38:21.104068 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:21.103974    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 10:38:21.104404 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:21.104386    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 10:38:21.104532 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:21.104391    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-d7gr5\""
Jul 28 10:38:21.104661 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:21.104641    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-1e74d8-kube-rbac-proxy-sar-config\""
Jul 28 10:38:21.111072 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:21.111050    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-1e74d8-predictor-6dfc9b49f6-2q5xs"]
Jul 28 10:38:21.160620 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:21.160587    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:38:21.238580 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:21.238543    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/346977e2-3969-46ac-b7c6-433d8472caf6-kserve-provision-location\") pod \"isvc-primary-1e74d8-predictor-6dfc9b49f6-2q5xs\" (UID: \"346977e2-3969-46ac-b7c6-433d8472caf6\") " pod="kserve-ci-e2e-test/isvc-primary-1e74d8-predictor-6dfc9b49f6-2q5xs"
Jul 28 10:38:21.238754 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:21.238594    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c2tq\" (UniqueName: \"kubernetes.io/projected/346977e2-3969-46ac-b7c6-433d8472caf6-kube-api-access-7c2tq\") pod \"isvc-primary-1e74d8-predictor-6dfc9b49f6-2q5xs\" (UID: \"346977e2-3969-46ac-b7c6-433d8472caf6\") " pod="kserve-ci-e2e-test/isvc-primary-1e74d8-predictor-6dfc9b49f6-2q5xs"
Jul 28 10:38:21.238754 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:21.238643    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/346977e2-3969-46ac-b7c6-433d8472caf6-kube-rbac-proxy-sar-config\") pod \"isvc-primary-1e74d8-predictor-6dfc9b49f6-2q5xs\" (UID: \"346977e2-3969-46ac-b7c6-433d8472caf6\") " pod="kserve-ci-e2e-test/isvc-primary-1e74d8-predictor-6dfc9b49f6-2q5xs"
Jul 28 10:38:21.238754 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:21.238694    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/346977e2-3969-46ac-b7c6-433d8472caf6-proxy-tls\") pod \"isvc-primary-1e74d8-predictor-6dfc9b49f6-2q5xs\" (UID: \"346977e2-3969-46ac-b7c6-433d8472caf6\") " pod="kserve-ci-e2e-test/isvc-primary-1e74d8-predictor-6dfc9b49f6-2q5xs"
Jul 28 10:38:21.238887 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:21.238775    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/346977e2-3969-46ac-b7c6-433d8472caf6-cabundle-cert\") pod \"isvc-primary-1e74d8-predictor-6dfc9b49f6-2q5xs\" (UID: \"346977e2-3969-46ac-b7c6-433d8472caf6\") " pod="kserve-ci-e2e-test/isvc-primary-1e74d8-predictor-6dfc9b49f6-2q5xs"
Jul 28 10:38:21.339404 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:21.339326    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/346977e2-3969-46ac-b7c6-433d8472caf6-cabundle-cert\") pod \"isvc-primary-1e74d8-predictor-6dfc9b49f6-2q5xs\" (UID: \"346977e2-3969-46ac-b7c6-433d8472caf6\") " pod="kserve-ci-e2e-test/isvc-primary-1e74d8-predictor-6dfc9b49f6-2q5xs"
Jul 28 10:38:21.339404 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:21.339376    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/346977e2-3969-46ac-b7c6-433d8472caf6-kserve-provision-location\") pod \"isvc-primary-1e74d8-predictor-6dfc9b49f6-2q5xs\" (UID: \"346977e2-3969-46ac-b7c6-433d8472caf6\") " pod="kserve-ci-e2e-test/isvc-primary-1e74d8-predictor-6dfc9b49f6-2q5xs"
Jul 28 10:38:21.339628 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:21.339416    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c2tq\" (UniqueName: \"kubernetes.io/projected/346977e2-3969-46ac-b7c6-433d8472caf6-kube-api-access-7c2tq\") pod \"isvc-primary-1e74d8-predictor-6dfc9b49f6-2q5xs\" (UID: \"346977e2-3969-46ac-b7c6-433d8472caf6\") " pod="kserve-ci-e2e-test/isvc-primary-1e74d8-predictor-6dfc9b49f6-2q5xs"
Jul 28 10:38:21.339628 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:21.339443    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/346977e2-3969-46ac-b7c6-433d8472caf6-kube-rbac-proxy-sar-config\") pod \"isvc-primary-1e74d8-predictor-6dfc9b49f6-2q5xs\" (UID: \"346977e2-3969-46ac-b7c6-433d8472caf6\") " pod="kserve-ci-e2e-test/isvc-primary-1e74d8-predictor-6dfc9b49f6-2q5xs"
Jul 28 10:38:21.339628 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:21.339483    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/346977e2-3969-46ac-b7c6-433d8472caf6-proxy-tls\") pod \"isvc-primary-1e74d8-predictor-6dfc9b49f6-2q5xs\" (UID: \"346977e2-3969-46ac-b7c6-433d8472caf6\") " pod="kserve-ci-e2e-test/isvc-primary-1e74d8-predictor-6dfc9b49f6-2q5xs"
Jul 28 10:38:21.339834 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:21.339813    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/346977e2-3969-46ac-b7c6-433d8472caf6-kserve-provision-location\") pod \"isvc-primary-1e74d8-predictor-6dfc9b49f6-2q5xs\" (UID: \"346977e2-3969-46ac-b7c6-433d8472caf6\") " pod="kserve-ci-e2e-test/isvc-primary-1e74d8-predictor-6dfc9b49f6-2q5xs"
Jul 28 10:38:21.340080 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:21.340058    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/346977e2-3969-46ac-b7c6-433d8472caf6-cabundle-cert\") pod \"isvc-primary-1e74d8-predictor-6dfc9b49f6-2q5xs\" (UID: \"346977e2-3969-46ac-b7c6-433d8472caf6\") " pod="kserve-ci-e2e-test/isvc-primary-1e74d8-predictor-6dfc9b49f6-2q5xs"
Jul 28 10:38:21.340161 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:21.340142    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/346977e2-3969-46ac-b7c6-433d8472caf6-kube-rbac-proxy-sar-config\") pod \"isvc-primary-1e74d8-predictor-6dfc9b49f6-2q5xs\" (UID: \"346977e2-3969-46ac-b7c6-433d8472caf6\") " pod="kserve-ci-e2e-test/isvc-primary-1e74d8-predictor-6dfc9b49f6-2q5xs"
Jul 28 10:38:21.341990 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:21.341969    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/346977e2-3969-46ac-b7c6-433d8472caf6-proxy-tls\") pod \"isvc-primary-1e74d8-predictor-6dfc9b49f6-2q5xs\" (UID: \"346977e2-3969-46ac-b7c6-433d8472caf6\") " pod="kserve-ci-e2e-test/isvc-primary-1e74d8-predictor-6dfc9b49f6-2q5xs"
Jul 28 10:38:21.348779 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:21.348749    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c2tq\" (UniqueName: \"kubernetes.io/projected/346977e2-3969-46ac-b7c6-433d8472caf6-kube-api-access-7c2tq\") pod \"isvc-primary-1e74d8-predictor-6dfc9b49f6-2q5xs\" (UID: \"346977e2-3969-46ac-b7c6-433d8472caf6\") " pod="kserve-ci-e2e-test/isvc-primary-1e74d8-predictor-6dfc9b49f6-2q5xs"
Jul 28 10:38:21.409014 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:21.408971    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-1e74d8-predictor-6dfc9b49f6-2q5xs"
Jul 28 10:38:21.541551 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:21.541527    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-1e74d8-predictor-6dfc9b49f6-2q5xs"]
Jul 28 10:38:21.543938 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:38:21.543913    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod346977e2_3969_46ac_b7c6_433d8472caf6.slice/crio-eaf44b072a88d0c26b8635befd65b7505816f729e1359cce28bec8cdd9d2d39d WatchSource:0}: Error finding container eaf44b072a88d0c26b8635befd65b7505816f729e1359cce28bec8cdd9d2d39d: Status 404 returned error can't find the container with id eaf44b072a88d0c26b8635befd65b7505816f729e1359cce28bec8cdd9d2d39d
Jul 28 10:38:21.689130 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:21.689096    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-1e74d8-predictor-6dfc9b49f6-2q5xs" event={"ID":"346977e2-3969-46ac-b7c6-433d8472caf6","Type":"ContainerStarted","Data":"f3d18dd9ca8d69226021eed2be2ce3fa4acd50cacfe4a9f7a19aa74824431408"}
Jul 28 10:38:21.689289 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:21.689158    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-1e74d8-predictor-6dfc9b49f6-2q5xs" event={"ID":"346977e2-3969-46ac-b7c6-433d8472caf6","Type":"ContainerStarted","Data":"eaf44b072a88d0c26b8635befd65b7505816f729e1359cce28bec8cdd9d2d39d"}
Jul 28 10:38:25.707515 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:25.707476    2583 generic.go:356] "Generic (PLEG): container finished" podID="346977e2-3969-46ac-b7c6-433d8472caf6" containerID="f3d18dd9ca8d69226021eed2be2ce3fa4acd50cacfe4a9f7a19aa74824431408" exitCode=0
Jul 28 10:38:25.707902 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:25.707540    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-1e74d8-predictor-6dfc9b49f6-2q5xs" event={"ID":"346977e2-3969-46ac-b7c6-433d8472caf6","Type":"ContainerDied","Data":"f3d18dd9ca8d69226021eed2be2ce3fa4acd50cacfe4a9f7a19aa74824431408"}
Jul 28 10:38:26.714458 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:26.714419    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-1e74d8-predictor-6dfc9b49f6-2q5xs" event={"ID":"346977e2-3969-46ac-b7c6-433d8472caf6","Type":"ContainerStarted","Data":"46e92b772838c436ea51b24297065ec12007df2a6a27d20d1af3cf4f1426e9e8"}
Jul 28 10:38:26.714944 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:26.714497    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-1e74d8-predictor-6dfc9b49f6-2q5xs" event={"ID":"346977e2-3969-46ac-b7c6-433d8472caf6","Type":"ContainerStarted","Data":"2bbc1ed965be775843215e43b6a078ead8d050c5b6eaf2a100a374a27da6e39d"}
Jul 28 10:38:26.714944 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:26.714712    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-1e74d8-predictor-6dfc9b49f6-2q5xs"
Jul 28 10:38:26.733199 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:26.733131    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-1e74d8-predictor-6dfc9b49f6-2q5xs" podStartSLOduration=1.7838492700000002 podStartE2EDuration="5.733116982s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.949267712s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:38:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:38:26.731658716 +0000 UTC m=+1845.832165270" watchObservedRunningTime="2026-07-28 10:38:26.733116982 +0000 UTC m=+1845.833623536"
Jul 28 10:38:27.719202 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:27.719156    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-1e74d8-predictor-6dfc9b49f6-2q5xs"
Jul 28 10:38:27.720309 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:27.720283    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-1e74d8-predictor-6dfc9b49f6-2q5xs" podUID="346977e2-3969-46ac-b7c6-433d8472caf6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 28 10:38:28.723454 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:28.723415    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-1e74d8-predictor-6dfc9b49f6-2q5xs" podUID="346977e2-3969-46ac-b7c6-433d8472caf6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 28 10:38:33.727961 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:33.727930    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-1e74d8-predictor-6dfc9b49f6-2q5xs"
Jul 28 10:38:33.730341 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:33.728557    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-1e74d8-predictor-6dfc9b49f6-2q5xs" podUID="346977e2-3969-46ac-b7c6-433d8472caf6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 28 10:38:43.729244 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:43.729197    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-1e74d8-predictor-6dfc9b49f6-2q5xs" podUID="346977e2-3969-46ac-b7c6-433d8472caf6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 28 10:38:53.729167 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:38:53.729125    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-1e74d8-predictor-6dfc9b49f6-2q5xs" podUID="346977e2-3969-46ac-b7c6-433d8472caf6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 28 10:39:03.728503 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:39:03.728402    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-1e74d8-predictor-6dfc9b49f6-2q5xs" podUID="346977e2-3969-46ac-b7c6-433d8472caf6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 28 10:39:13.729152 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:39:13.729116    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-1e74d8-predictor-6dfc9b49f6-2q5xs" podUID="346977e2-3969-46ac-b7c6-433d8472caf6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 28 10:39:23.729047 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:39:23.728996    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-1e74d8-predictor-6dfc9b49f6-2q5xs" podUID="346977e2-3969-46ac-b7c6-433d8472caf6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 28 10:39:33.728622 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:39:33.728570    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-1e74d8-predictor-6dfc9b49f6-2q5xs" podUID="346977e2-3969-46ac-b7c6-433d8472caf6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 28 10:39:43.729845 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:39:43.729810    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-1e74d8-predictor-6dfc9b49f6-2q5xs"
Jul 28 10:40:06.255307 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:06.255264    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:40:06.313725 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:06.313696    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-1e74d8-predictor-7f998fd97f-sbv9t"]
Jul 28 10:40:06.317235 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:06.317220    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-1e74d8-predictor-7f998fd97f-sbv9t"
Jul 28 10:40:06.321302 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:06.321279    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-1e74d8-dockercfg-h8q2c\""
Jul 28 10:40:06.321755 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:06.321737    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-1e74d8-predictor-serving-cert\""
Jul 28 10:40:06.321832 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:06.321797    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-1e74d8-kube-rbac-proxy-sar-config\""
Jul 28 10:40:06.331314 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:06.331298    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-1e74d8\""
Jul 28 10:40:06.338660 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:06.338634    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-1e74d8-predictor-7f998fd97f-sbv9t"]
Jul 28 10:40:06.418993 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:06.418964    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/33eb3664-13f0-4c97-933f-d263733467f1-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-1e74d8-predictor-7f998fd97f-sbv9t\" (UID: \"33eb3664-13f0-4c97-933f-d263733467f1\") " pod="kserve-ci-e2e-test/isvc-secondary-1e74d8-predictor-7f998fd97f-sbv9t"
Jul 28 10:40:06.419083 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:06.419002    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/33eb3664-13f0-4c97-933f-d263733467f1-proxy-tls\") pod \"isvc-secondary-1e74d8-predictor-7f998fd97f-sbv9t\" (UID: \"33eb3664-13f0-4c97-933f-d263733467f1\") " pod="kserve-ci-e2e-test/isvc-secondary-1e74d8-predictor-7f998fd97f-sbv9t"
Jul 28 10:40:06.419133 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:06.419079    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j79ws\" (UniqueName: \"kubernetes.io/projected/33eb3664-13f0-4c97-933f-d263733467f1-kube-api-access-j79ws\") pod \"isvc-secondary-1e74d8-predictor-7f998fd97f-sbv9t\" (UID: \"33eb3664-13f0-4c97-933f-d263733467f1\") " pod="kserve-ci-e2e-test/isvc-secondary-1e74d8-predictor-7f998fd97f-sbv9t"
Jul 28 10:40:06.419133 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:06.419118    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33eb3664-13f0-4c97-933f-d263733467f1-kserve-provision-location\") pod \"isvc-secondary-1e74d8-predictor-7f998fd97f-sbv9t\" (UID: \"33eb3664-13f0-4c97-933f-d263733467f1\") " pod="kserve-ci-e2e-test/isvc-secondary-1e74d8-predictor-7f998fd97f-sbv9t"
Jul 28 10:40:06.419242 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:06.419156    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/33eb3664-13f0-4c97-933f-d263733467f1-cabundle-cert\") pod \"isvc-secondary-1e74d8-predictor-7f998fd97f-sbv9t\" (UID: \"33eb3664-13f0-4c97-933f-d263733467f1\") " pod="kserve-ci-e2e-test/isvc-secondary-1e74d8-predictor-7f998fd97f-sbv9t"
Jul 28 10:40:06.520125 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:06.520069    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/33eb3664-13f0-4c97-933f-d263733467f1-cabundle-cert\") pod \"isvc-secondary-1e74d8-predictor-7f998fd97f-sbv9t\" (UID: \"33eb3664-13f0-4c97-933f-d263733467f1\") " pod="kserve-ci-e2e-test/isvc-secondary-1e74d8-predictor-7f998fd97f-sbv9t"
Jul 28 10:40:06.520125 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:06.520116    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/33eb3664-13f0-4c97-933f-d263733467f1-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-1e74d8-predictor-7f998fd97f-sbv9t\" (UID: \"33eb3664-13f0-4c97-933f-d263733467f1\") " pod="kserve-ci-e2e-test/isvc-secondary-1e74d8-predictor-7f998fd97f-sbv9t"
Jul 28 10:40:06.520298 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:06.520143    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/33eb3664-13f0-4c97-933f-d263733467f1-proxy-tls\") pod \"isvc-secondary-1e74d8-predictor-7f998fd97f-sbv9t\" (UID: \"33eb3664-13f0-4c97-933f-d263733467f1\") " pod="kserve-ci-e2e-test/isvc-secondary-1e74d8-predictor-7f998fd97f-sbv9t"
Jul 28 10:40:06.520298 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:06.520210    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j79ws\" (UniqueName: \"kubernetes.io/projected/33eb3664-13f0-4c97-933f-d263733467f1-kube-api-access-j79ws\") pod \"isvc-secondary-1e74d8-predictor-7f998fd97f-sbv9t\" (UID: \"33eb3664-13f0-4c97-933f-d263733467f1\") " pod="kserve-ci-e2e-test/isvc-secondary-1e74d8-predictor-7f998fd97f-sbv9t"
Jul 28 10:40:06.520298 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:06.520255    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33eb3664-13f0-4c97-933f-d263733467f1-kserve-provision-location\") pod \"isvc-secondary-1e74d8-predictor-7f998fd97f-sbv9t\" (UID: \"33eb3664-13f0-4c97-933f-d263733467f1\") " pod="kserve-ci-e2e-test/isvc-secondary-1e74d8-predictor-7f998fd97f-sbv9t"
Jul 28 10:40:06.520456 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:40:06.520332    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-secondary-1e74d8-predictor-serving-cert: secret "isvc-secondary-1e74d8-predictor-serving-cert" not found
Jul 28 10:40:06.520456 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:40:06.520408    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33eb3664-13f0-4c97-933f-d263733467f1-proxy-tls podName:33eb3664-13f0-4c97-933f-d263733467f1 nodeName:}" failed. No retries permitted until 2026-07-28 10:40:07.020386378 +0000 UTC m=+1946.120892923 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/33eb3664-13f0-4c97-933f-d263733467f1-proxy-tls") pod "isvc-secondary-1e74d8-predictor-7f998fd97f-sbv9t" (UID: "33eb3664-13f0-4c97-933f-d263733467f1") : secret "isvc-secondary-1e74d8-predictor-serving-cert" not found
Jul 28 10:40:06.520675 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:06.520654    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33eb3664-13f0-4c97-933f-d263733467f1-kserve-provision-location\") pod \"isvc-secondary-1e74d8-predictor-7f998fd97f-sbv9t\" (UID: \"33eb3664-13f0-4c97-933f-d263733467f1\") " pod="kserve-ci-e2e-test/isvc-secondary-1e74d8-predictor-7f998fd97f-sbv9t"
Jul 28 10:40:06.520794 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:06.520777    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/33eb3664-13f0-4c97-933f-d263733467f1-cabundle-cert\") pod \"isvc-secondary-1e74d8-predictor-7f998fd97f-sbv9t\" (UID: \"33eb3664-13f0-4c97-933f-d263733467f1\") " pod="kserve-ci-e2e-test/isvc-secondary-1e74d8-predictor-7f998fd97f-sbv9t"
Jul 28 10:40:06.520794 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:06.520788    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/33eb3664-13f0-4c97-933f-d263733467f1-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-1e74d8-predictor-7f998fd97f-sbv9t\" (UID: \"33eb3664-13f0-4c97-933f-d263733467f1\") " pod="kserve-ci-e2e-test/isvc-secondary-1e74d8-predictor-7f998fd97f-sbv9t"
Jul 28 10:40:06.529254 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:06.529234    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j79ws\" (UniqueName: \"kubernetes.io/projected/33eb3664-13f0-4c97-933f-d263733467f1-kube-api-access-j79ws\") pod \"isvc-secondary-1e74d8-predictor-7f998fd97f-sbv9t\" (UID: \"33eb3664-13f0-4c97-933f-d263733467f1\") " pod="kserve-ci-e2e-test/isvc-secondary-1e74d8-predictor-7f998fd97f-sbv9t"
Jul 28 10:40:06.580437 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:06.580415    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:40:07.024820 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:07.024783    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/33eb3664-13f0-4c97-933f-d263733467f1-proxy-tls\") pod \"isvc-secondary-1e74d8-predictor-7f998fd97f-sbv9t\" (UID: \"33eb3664-13f0-4c97-933f-d263733467f1\") " pod="kserve-ci-e2e-test/isvc-secondary-1e74d8-predictor-7f998fd97f-sbv9t"
Jul 28 10:40:07.027419 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:07.027386    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/33eb3664-13f0-4c97-933f-d263733467f1-proxy-tls\") pod \"isvc-secondary-1e74d8-predictor-7f998fd97f-sbv9t\" (UID: \"33eb3664-13f0-4c97-933f-d263733467f1\") " pod="kserve-ci-e2e-test/isvc-secondary-1e74d8-predictor-7f998fd97f-sbv9t"
Jul 28 10:40:07.227280 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:07.227246    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-1e74d8-predictor-7f998fd97f-sbv9t"
Jul 28 10:40:07.353824 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:07.353795    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-1e74d8-predictor-7f998fd97f-sbv9t"]
Jul 28 10:40:07.355686 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:40:07.355657    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33eb3664_13f0_4c97_933f_d263733467f1.slice/crio-3b63f91b710e2601fbd3cc912289f8d99a5fe1ccc301f33387081773cebbbe28 WatchSource:0}: Error finding container 3b63f91b710e2601fbd3cc912289f8d99a5fe1ccc301f33387081773cebbbe28: Status 404 returned error can't find the container with id 3b63f91b710e2601fbd3cc912289f8d99a5fe1ccc301f33387081773cebbbe28
Jul 28 10:40:08.176566 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:08.176533    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-1e74d8-predictor-7f998fd97f-sbv9t" event={"ID":"33eb3664-13f0-4c97-933f-d263733467f1","Type":"ContainerStarted","Data":"12ce47ea6bdd124c5b1d2df7e302c882589d23922c6e1b7b81a60313f3f5a36e"}
Jul 28 10:40:08.176718 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:08.176587    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-1e74d8-predictor-7f998fd97f-sbv9t" event={"ID":"33eb3664-13f0-4c97-933f-d263733467f1","Type":"ContainerStarted","Data":"3b63f91b710e2601fbd3cc912289f8d99a5fe1ccc301f33387081773cebbbe28"}
Jul 28 10:40:13.202486 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:13.202452    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-1e74d8-predictor-7f998fd97f-sbv9t_33eb3664-13f0-4c97-933f-d263733467f1/storage-initializer/0.log"
Jul 28 10:40:13.202865 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:13.202490    2583 generic.go:356] "Generic (PLEG): container finished" podID="33eb3664-13f0-4c97-933f-d263733467f1" containerID="12ce47ea6bdd124c5b1d2df7e302c882589d23922c6e1b7b81a60313f3f5a36e" exitCode=1
Jul 28 10:40:13.202865 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:13.202532    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-1e74d8-predictor-7f998fd97f-sbv9t" event={"ID":"33eb3664-13f0-4c97-933f-d263733467f1","Type":"ContainerDied","Data":"12ce47ea6bdd124c5b1d2df7e302c882589d23922c6e1b7b81a60313f3f5a36e"}
Jul 28 10:40:14.209288 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:14.209256    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-1e74d8-predictor-7f998fd97f-sbv9t_33eb3664-13f0-4c97-933f-d263733467f1/storage-initializer/0.log"
Jul 28 10:40:14.209687 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:14.209381    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-1e74d8-predictor-7f998fd97f-sbv9t" event={"ID":"33eb3664-13f0-4c97-933f-d263733467f1","Type":"ContainerStarted","Data":"9941112270b7f4b4c6f709d218791c0bf1feed7e0f6c8fe4999e7675124e6ee6"}
Jul 28 10:40:17.224680 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:17.224654    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-1e74d8-predictor-7f998fd97f-sbv9t_33eb3664-13f0-4c97-933f-d263733467f1/storage-initializer/1.log"
Jul 28 10:40:17.225080 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:17.225016    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-1e74d8-predictor-7f998fd97f-sbv9t_33eb3664-13f0-4c97-933f-d263733467f1/storage-initializer/0.log"
Jul 28 10:40:17.225080 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:17.225047    2583 generic.go:356] "Generic (PLEG): container finished" podID="33eb3664-13f0-4c97-933f-d263733467f1" containerID="9941112270b7f4b4c6f709d218791c0bf1feed7e0f6c8fe4999e7675124e6ee6" exitCode=1
Jul 28 10:40:17.225154 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:17.225093    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-1e74d8-predictor-7f998fd97f-sbv9t" event={"ID":"33eb3664-13f0-4c97-933f-d263733467f1","Type":"ContainerDied","Data":"9941112270b7f4b4c6f709d218791c0bf1feed7e0f6c8fe4999e7675124e6ee6"}
Jul 28 10:40:17.225214 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:17.225154    2583 scope.go:117] "RemoveContainer" containerID="12ce47ea6bdd124c5b1d2df7e302c882589d23922c6e1b7b81a60313f3f5a36e"
Jul 28 10:40:17.225506 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:17.225483    2583 scope.go:117] "RemoveContainer" containerID="12ce47ea6bdd124c5b1d2df7e302c882589d23922c6e1b7b81a60313f3f5a36e"
Jul 28 10:40:17.236355 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:40:17.236321    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-1e74d8-predictor-7f998fd97f-sbv9t_kserve-ci-e2e-test(33eb3664-13f0-4c97-933f-d263733467f1)\"" pod="kserve-ci-e2e-test/isvc-secondary-1e74d8-predictor-7f998fd97f-sbv9t" podUID="33eb3664-13f0-4c97-933f-d263733467f1"
Jul 28 10:40:17.236600 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:40:17.236574    2583 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-1e74d8-predictor-7f998fd97f-sbv9t_kserve-ci-e2e-test_33eb3664-13f0-4c97-933f-d263733467f1_0 in pod sandbox 3b63f91b710e2601fbd3cc912289f8d99a5fe1ccc301f33387081773cebbbe28 from index: no such id: '12ce47ea6bdd124c5b1d2df7e302c882589d23922c6e1b7b81a60313f3f5a36e'" containerID="12ce47ea6bdd124c5b1d2df7e302c882589d23922c6e1b7b81a60313f3f5a36e"
Jul 28 10:40:17.236662 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:17.236611    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12ce47ea6bdd124c5b1d2df7e302c882589d23922c6e1b7b81a60313f3f5a36e"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-1e74d8-predictor-7f998fd97f-sbv9t_kserve-ci-e2e-test_33eb3664-13f0-4c97-933f-d263733467f1_0 in pod sandbox 3b63f91b710e2601fbd3cc912289f8d99a5fe1ccc301f33387081773cebbbe28 from index: no such id: '12ce47ea6bdd124c5b1d2df7e302c882589d23922c6e1b7b81a60313f3f5a36e'"
Jul 28 10:40:18.230570 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:18.230544    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-1e74d8-predictor-7f998fd97f-sbv9t_33eb3664-13f0-4c97-933f-d263733467f1/storage-initializer/1.log"
Jul 28 10:40:24.304676 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:24.304641    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-1e74d8-predictor-7f998fd97f-sbv9t"]
Jul 28 10:40:24.324818 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:24.324792    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:40:24.352976 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:24.352945    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-1e74d8-predictor-6dfc9b49f6-2q5xs"]
Jul 28 10:40:24.353245 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:24.353199    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-1e74d8-predictor-6dfc9b49f6-2q5xs" podUID="346977e2-3969-46ac-b7c6-433d8472caf6" containerName="kserve-container" containerID="cri-o://2bbc1ed965be775843215e43b6a078ead8d050c5b6eaf2a100a374a27da6e39d" gracePeriod=30
Jul 28 10:40:24.353245 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:24.353230    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-1e74d8-predictor-6dfc9b49f6-2q5xs" podUID="346977e2-3969-46ac-b7c6-433d8472caf6" containerName="kube-rbac-proxy" containerID="cri-o://46e92b772838c436ea51b24297065ec12007df2a6a27d20d1af3cf4f1426e9e8" gracePeriod=30
Jul 28 10:40:24.382821 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:24.382790    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:40:24.472884 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:24.472863    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-1e74d8-predictor-7f998fd97f-sbv9t_33eb3664-13f0-4c97-933f-d263733467f1/storage-initializer/1.log"
Jul 28 10:40:24.475111 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:24.475095    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-1e74d8-predictor-7f998fd97f-sbv9t"
Jul 28 10:40:24.657019 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:24.656999    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j79ws\" (UniqueName: \"kubernetes.io/projected/33eb3664-13f0-4c97-933f-d263733467f1-kube-api-access-j79ws\") pod \"33eb3664-13f0-4c97-933f-d263733467f1\" (UID: \"33eb3664-13f0-4c97-933f-d263733467f1\") "
Jul 28 10:40:24.657137 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:24.657059    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/33eb3664-13f0-4c97-933f-d263733467f1-kube-rbac-proxy-sar-config\") pod \"33eb3664-13f0-4c97-933f-d263733467f1\" (UID: \"33eb3664-13f0-4c97-933f-d263733467f1\") "
Jul 28 10:40:24.657137 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:24.657076    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/33eb3664-13f0-4c97-933f-d263733467f1-cabundle-cert\") pod \"33eb3664-13f0-4c97-933f-d263733467f1\" (UID: \"33eb3664-13f0-4c97-933f-d263733467f1\") "
Jul 28 10:40:24.657267 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:24.657248    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33eb3664-13f0-4c97-933f-d263733467f1-kserve-provision-location\") pod \"33eb3664-13f0-4c97-933f-d263733467f1\" (UID: \"33eb3664-13f0-4c97-933f-d263733467f1\") "
Jul 28 10:40:24.657372 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:24.657357    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/33eb3664-13f0-4c97-933f-d263733467f1-proxy-tls\") pod \"33eb3664-13f0-4c97-933f-d263733467f1\" (UID: \"33eb3664-13f0-4c97-933f-d263733467f1\") "
Jul 28 10:40:24.657532 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:24.657499    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33eb3664-13f0-4c97-933f-d263733467f1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "33eb3664-13f0-4c97-933f-d263733467f1" (UID: "33eb3664-13f0-4c97-933f-d263733467f1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:40:24.657532 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:24.657518    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33eb3664-13f0-4c97-933f-d263733467f1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "33eb3664-13f0-4c97-933f-d263733467f1" (UID: "33eb3664-13f0-4c97-933f-d263733467f1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:40:24.657663 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:24.657508    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33eb3664-13f0-4c97-933f-d263733467f1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "33eb3664-13f0-4c97-933f-d263733467f1" (UID: "33eb3664-13f0-4c97-933f-d263733467f1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:40:24.657663 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:24.657624    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/33eb3664-13f0-4c97-933f-d263733467f1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:40:24.657663 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:24.657637    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/33eb3664-13f0-4c97-933f-d263733467f1-cabundle-cert\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:40:24.657663 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:24.657647    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33eb3664-13f0-4c97-933f-d263733467f1-kserve-provision-location\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:40:24.659235 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:24.659208    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33eb3664-13f0-4c97-933f-d263733467f1-kube-api-access-j79ws" (OuterVolumeSpecName: "kube-api-access-j79ws") pod "33eb3664-13f0-4c97-933f-d263733467f1" (UID: "33eb3664-13f0-4c97-933f-d263733467f1"). InnerVolumeSpecName "kube-api-access-j79ws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:40:24.659235 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:24.659214    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33eb3664-13f0-4c97-933f-d263733467f1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "33eb3664-13f0-4c97-933f-d263733467f1" (UID: "33eb3664-13f0-4c97-933f-d263733467f1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:40:24.757958 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:24.757935    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/33eb3664-13f0-4c97-933f-d263733467f1-proxy-tls\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:40:24.757958 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:24.757955    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j79ws\" (UniqueName: \"kubernetes.io/projected/33eb3664-13f0-4c97-933f-d263733467f1-kube-api-access-j79ws\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:40:25.265438 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:25.265401    2583 generic.go:356] "Generic (PLEG): container finished" podID="346977e2-3969-46ac-b7c6-433d8472caf6" containerID="46e92b772838c436ea51b24297065ec12007df2a6a27d20d1af3cf4f1426e9e8" exitCode=2
Jul 28 10:40:25.265618 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:25.265471    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-1e74d8-predictor-6dfc9b49f6-2q5xs" event={"ID":"346977e2-3969-46ac-b7c6-433d8472caf6","Type":"ContainerDied","Data":"46e92b772838c436ea51b24297065ec12007df2a6a27d20d1af3cf4f1426e9e8"}
Jul 28 10:40:25.266697 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:25.266682    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-1e74d8-predictor-7f998fd97f-sbv9t_33eb3664-13f0-4c97-933f-d263733467f1/storage-initializer/1.log"
Jul 28 10:40:25.266780 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:25.266719    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-1e74d8-predictor-7f998fd97f-sbv9t" event={"ID":"33eb3664-13f0-4c97-933f-d263733467f1","Type":"ContainerDied","Data":"3b63f91b710e2601fbd3cc912289f8d99a5fe1ccc301f33387081773cebbbe28"}
Jul 28 10:40:25.266780 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:25.266762    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-1e74d8-predictor-7f998fd97f-sbv9t"
Jul 28 10:40:25.266780 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:25.266770    2583 scope.go:117] "RemoveContainer" containerID="9941112270b7f4b4c6f709d218791c0bf1feed7e0f6c8fe4999e7675124e6ee6"
Jul 28 10:40:25.304428 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:25.304405    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-1e74d8-predictor-7f998fd97f-sbv9t"]
Jul 28 10:40:25.310250 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:25.310232    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-1e74d8-predictor-7f998fd97f-sbv9t"]
Jul 28 10:40:25.438410 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:25.438385    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33eb3664-13f0-4c97-933f-d263733467f1" path="/var/lib/kubelet/pods/33eb3664-13f0-4c97-933f-d263733467f1/volumes"
Jul 28 10:40:28.281923 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:28.281896    2583 generic.go:356] "Generic (PLEG): container finished" podID="346977e2-3969-46ac-b7c6-433d8472caf6" containerID="2bbc1ed965be775843215e43b6a078ead8d050c5b6eaf2a100a374a27da6e39d" exitCode=0
Jul 28 10:40:28.282339 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:28.281946    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-1e74d8-predictor-6dfc9b49f6-2q5xs" event={"ID":"346977e2-3969-46ac-b7c6-433d8472caf6","Type":"ContainerDied","Data":"2bbc1ed965be775843215e43b6a078ead8d050c5b6eaf2a100a374a27da6e39d"}
Jul 28 10:40:28.403392 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:28.403368    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-1e74d8-predictor-6dfc9b49f6-2q5xs"
Jul 28 10:40:28.481398 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:28.481369    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/346977e2-3969-46ac-b7c6-433d8472caf6-kserve-provision-location\") pod \"346977e2-3969-46ac-b7c6-433d8472caf6\" (UID: \"346977e2-3969-46ac-b7c6-433d8472caf6\") "
Jul 28 10:40:28.481527 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:28.481408    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/346977e2-3969-46ac-b7c6-433d8472caf6-kube-rbac-proxy-sar-config\") pod \"346977e2-3969-46ac-b7c6-433d8472caf6\" (UID: \"346977e2-3969-46ac-b7c6-433d8472caf6\") "
Jul 28 10:40:28.481527 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:28.481447    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7c2tq\" (UniqueName: \"kubernetes.io/projected/346977e2-3969-46ac-b7c6-433d8472caf6-kube-api-access-7c2tq\") pod \"346977e2-3969-46ac-b7c6-433d8472caf6\" (UID: \"346977e2-3969-46ac-b7c6-433d8472caf6\") "
Jul 28 10:40:28.481527 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:28.481475    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/346977e2-3969-46ac-b7c6-433d8472caf6-proxy-tls\") pod \"346977e2-3969-46ac-b7c6-433d8472caf6\" (UID: \"346977e2-3969-46ac-b7c6-433d8472caf6\") "
Jul 28 10:40:28.481527 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:28.481499    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/346977e2-3969-46ac-b7c6-433d8472caf6-cabundle-cert\") pod \"346977e2-3969-46ac-b7c6-433d8472caf6\" (UID: \"346977e2-3969-46ac-b7c6-433d8472caf6\") "
Jul 28 10:40:28.481765 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:28.481710    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/346977e2-3969-46ac-b7c6-433d8472caf6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "346977e2-3969-46ac-b7c6-433d8472caf6" (UID: "346977e2-3969-46ac-b7c6-433d8472caf6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:40:28.481838 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:28.481745    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/346977e2-3969-46ac-b7c6-433d8472caf6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "346977e2-3969-46ac-b7c6-433d8472caf6" (UID: "346977e2-3969-46ac-b7c6-433d8472caf6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:40:28.481917 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:28.481897    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/346977e2-3969-46ac-b7c6-433d8472caf6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "346977e2-3969-46ac-b7c6-433d8472caf6" (UID: "346977e2-3969-46ac-b7c6-433d8472caf6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:40:28.483472 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:28.483452    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/346977e2-3969-46ac-b7c6-433d8472caf6-kube-api-access-7c2tq" (OuterVolumeSpecName: "kube-api-access-7c2tq") pod "346977e2-3969-46ac-b7c6-433d8472caf6" (UID: "346977e2-3969-46ac-b7c6-433d8472caf6"). InnerVolumeSpecName "kube-api-access-7c2tq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:40:28.483562 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:28.483500    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/346977e2-3969-46ac-b7c6-433d8472caf6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "346977e2-3969-46ac-b7c6-433d8472caf6" (UID: "346977e2-3969-46ac-b7c6-433d8472caf6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:40:28.582853 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:28.582833    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/346977e2-3969-46ac-b7c6-433d8472caf6-kserve-provision-location\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:40:28.582853 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:28.582853    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/346977e2-3969-46ac-b7c6-433d8472caf6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:40:28.583001 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:28.582864    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7c2tq\" (UniqueName: \"kubernetes.io/projected/346977e2-3969-46ac-b7c6-433d8472caf6-kube-api-access-7c2tq\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:40:28.583001 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:28.582873    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/346977e2-3969-46ac-b7c6-433d8472caf6-proxy-tls\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:40:28.583001 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:28.582882    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/346977e2-3969-46ac-b7c6-433d8472caf6-cabundle-cert\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:40:29.288962 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:29.288935    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-1e74d8-predictor-6dfc9b49f6-2q5xs"
Jul 28 10:40:29.289490 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:29.288932    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-1e74d8-predictor-6dfc9b49f6-2q5xs" event={"ID":"346977e2-3969-46ac-b7c6-433d8472caf6","Type":"ContainerDied","Data":"eaf44b072a88d0c26b8635befd65b7505816f729e1359cce28bec8cdd9d2d39d"}
Jul 28 10:40:29.289490 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:29.289093    2583 scope.go:117] "RemoveContainer" containerID="46e92b772838c436ea51b24297065ec12007df2a6a27d20d1af3cf4f1426e9e8"
Jul 28 10:40:29.299065 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:29.299045    2583 scope.go:117] "RemoveContainer" containerID="2bbc1ed965be775843215e43b6a078ead8d050c5b6eaf2a100a374a27da6e39d"
Jul 28 10:40:29.306988 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:29.306968    2583 scope.go:117] "RemoveContainer" containerID="f3d18dd9ca8d69226021eed2be2ce3fa4acd50cacfe4a9f7a19aa74824431408"
Jul 28 10:40:29.312127 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:29.312103    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-1e74d8-predictor-6dfc9b49f6-2q5xs"]
Jul 28 10:40:29.316685 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:29.316663    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-1e74d8-predictor-6dfc9b49f6-2q5xs"]
Jul 28 10:40:29.439749 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:29.439725    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="346977e2-3969-46ac-b7c6-433d8472caf6" path="/var/lib/kubelet/pods/346977e2-3969-46ac-b7c6-433d8472caf6/volumes"
Jul 28 10:40:29.516622 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:29.516594    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:40:29.548316 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:29.548256    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-505b2f-predictor-5df49d5965-gptlx"]
Jul 28 10:40:29.548722 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:29.548708    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="346977e2-3969-46ac-b7c6-433d8472caf6" containerName="kserve-container"
Jul 28 10:40:29.548769 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:29.548726    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="346977e2-3969-46ac-b7c6-433d8472caf6" containerName="kserve-container"
Jul 28 10:40:29.548769 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:29.548738    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="346977e2-3969-46ac-b7c6-433d8472caf6" containerName="kube-rbac-proxy"
Jul 28 10:40:29.548769 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:29.548746    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="346977e2-3969-46ac-b7c6-433d8472caf6" containerName="kube-rbac-proxy"
Jul 28 10:40:29.548769 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:29.548763    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33eb3664-13f0-4c97-933f-d263733467f1" containerName="storage-initializer"
Jul 28 10:40:29.548905 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:29.548772    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="33eb3664-13f0-4c97-933f-d263733467f1" containerName="storage-initializer"
Jul 28 10:40:29.548905 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:29.548782    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33eb3664-13f0-4c97-933f-d263733467f1" containerName="storage-initializer"
Jul 28 10:40:29.548905 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:29.548789    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="33eb3664-13f0-4c97-933f-d263733467f1" containerName="storage-initializer"
Jul 28 10:40:29.548905 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:29.548805    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="346977e2-3969-46ac-b7c6-433d8472caf6" containerName="storage-initializer"
Jul 28 10:40:29.548905 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:29.548813    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="346977e2-3969-46ac-b7c6-433d8472caf6" containerName="storage-initializer"
Jul 28 10:40:29.548905 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:29.548888    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="33eb3664-13f0-4c97-933f-d263733467f1" containerName="storage-initializer"
Jul 28 10:40:29.548905 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:29.548900    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="33eb3664-13f0-4c97-933f-d263733467f1" containerName="storage-initializer"
Jul 28 10:40:29.549139 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:29.548911    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="346977e2-3969-46ac-b7c6-433d8472caf6" containerName="kserve-container"
Jul 28 10:40:29.549139 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:29.548920    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="346977e2-3969-46ac-b7c6-433d8472caf6" containerName="kube-rbac-proxy"
Jul 28 10:40:29.553889 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:29.553872    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-505b2f-predictor-5df49d5965-gptlx"
Jul 28 10:40:29.556081 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:29.556061    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-505b2f-predictor-serving-cert\""
Jul 28 10:40:29.556358 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:29.556303    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-505b2f-kube-rbac-proxy-sar-config\""
Jul 28 10:40:29.556457 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:29.556381    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-505b2f-dockercfg-v7dfk\""
Jul 28 10:40:29.556457 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:29.556434    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 10:40:29.556676 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:29.556656    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 10:40:29.556785 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:29.556678    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-505b2f\""
Jul 28 10:40:29.556853 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:29.556815    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 10:40:29.563965 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:29.563942    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-505b2f-predictor-5df49d5965-gptlx"]
Jul 28 10:40:29.589767 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:29.589742    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef05ab63-fabf-4d18-8221-5f250c8e78e7-proxy-tls\") pod \"isvc-init-fail-505b2f-predictor-5df49d5965-gptlx\" (UID: \"ef05ab63-fabf-4d18-8221-5f250c8e78e7\") " pod="kserve-ci-e2e-test/isvc-init-fail-505b2f-predictor-5df49d5965-gptlx"
Jul 28 10:40:29.589883 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:29.589768    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ef05ab63-fabf-4d18-8221-5f250c8e78e7-cabundle-cert\") pod \"isvc-init-fail-505b2f-predictor-5df49d5965-gptlx\" (UID: \"ef05ab63-fabf-4d18-8221-5f250c8e78e7\") " pod="kserve-ci-e2e-test/isvc-init-fail-505b2f-predictor-5df49d5965-gptlx"
Jul 28 10:40:29.589883 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:29.589814    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef05ab63-fabf-4d18-8221-5f250c8e78e7-kserve-provision-location\") pod \"isvc-init-fail-505b2f-predictor-5df49d5965-gptlx\" (UID: \"ef05ab63-fabf-4d18-8221-5f250c8e78e7\") " pod="kserve-ci-e2e-test/isvc-init-fail-505b2f-predictor-5df49d5965-gptlx"
Jul 28 10:40:29.589883 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:29.589856    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef05ab63-fabf-4d18-8221-5f250c8e78e7-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-505b2f-predictor-5df49d5965-gptlx\" (UID: \"ef05ab63-fabf-4d18-8221-5f250c8e78e7\") " pod="kserve-ci-e2e-test/isvc-init-fail-505b2f-predictor-5df49d5965-gptlx"
Jul 28 10:40:29.590055 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:29.589897    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg2cp\" (UniqueName: \"kubernetes.io/projected/ef05ab63-fabf-4d18-8221-5f250c8e78e7-kube-api-access-mg2cp\") pod \"isvc-init-fail-505b2f-predictor-5df49d5965-gptlx\" (UID: \"ef05ab63-fabf-4d18-8221-5f250c8e78e7\") " pod="kserve-ci-e2e-test/isvc-init-fail-505b2f-predictor-5df49d5965-gptlx"
Jul 28 10:40:29.690268 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:29.690240    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef05ab63-fabf-4d18-8221-5f250c8e78e7-kserve-provision-location\") pod \"isvc-init-fail-505b2f-predictor-5df49d5965-gptlx\" (UID: \"ef05ab63-fabf-4d18-8221-5f250c8e78e7\") " pod="kserve-ci-e2e-test/isvc-init-fail-505b2f-predictor-5df49d5965-gptlx"
Jul 28 10:40:29.690375 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:29.690274    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef05ab63-fabf-4d18-8221-5f250c8e78e7-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-505b2f-predictor-5df49d5965-gptlx\" (UID: \"ef05ab63-fabf-4d18-8221-5f250c8e78e7\") " pod="kserve-ci-e2e-test/isvc-init-fail-505b2f-predictor-5df49d5965-gptlx"
Jul 28 10:40:29.690375 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:29.690291    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg2cp\" (UniqueName: \"kubernetes.io/projected/ef05ab63-fabf-4d18-8221-5f250c8e78e7-kube-api-access-mg2cp\") pod \"isvc-init-fail-505b2f-predictor-5df49d5965-gptlx\" (UID: \"ef05ab63-fabf-4d18-8221-5f250c8e78e7\") " pod="kserve-ci-e2e-test/isvc-init-fail-505b2f-predictor-5df49d5965-gptlx"
Jul 28 10:40:29.690375 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:29.690320    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef05ab63-fabf-4d18-8221-5f250c8e78e7-proxy-tls\") pod \"isvc-init-fail-505b2f-predictor-5df49d5965-gptlx\" (UID: \"ef05ab63-fabf-4d18-8221-5f250c8e78e7\") " pod="kserve-ci-e2e-test/isvc-init-fail-505b2f-predictor-5df49d5965-gptlx"
Jul 28 10:40:29.690375 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:29.690338    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ef05ab63-fabf-4d18-8221-5f250c8e78e7-cabundle-cert\") pod \"isvc-init-fail-505b2f-predictor-5df49d5965-gptlx\" (UID: \"ef05ab63-fabf-4d18-8221-5f250c8e78e7\") " pod="kserve-ci-e2e-test/isvc-init-fail-505b2f-predictor-5df49d5965-gptlx"
Jul 28 10:40:29.690656 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:29.690636    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef05ab63-fabf-4d18-8221-5f250c8e78e7-kserve-provision-location\") pod \"isvc-init-fail-505b2f-predictor-5df49d5965-gptlx\" (UID: \"ef05ab63-fabf-4d18-8221-5f250c8e78e7\") " pod="kserve-ci-e2e-test/isvc-init-fail-505b2f-predictor-5df49d5965-gptlx"
Jul 28 10:40:29.690910 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:29.690891    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ef05ab63-fabf-4d18-8221-5f250c8e78e7-cabundle-cert\") pod \"isvc-init-fail-505b2f-predictor-5df49d5965-gptlx\" (UID: \"ef05ab63-fabf-4d18-8221-5f250c8e78e7\") " pod="kserve-ci-e2e-test/isvc-init-fail-505b2f-predictor-5df49d5965-gptlx"
Jul 28 10:40:29.690960 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:29.690944    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef05ab63-fabf-4d18-8221-5f250c8e78e7-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-505b2f-predictor-5df49d5965-gptlx\" (UID: \"ef05ab63-fabf-4d18-8221-5f250c8e78e7\") " pod="kserve-ci-e2e-test/isvc-init-fail-505b2f-predictor-5df49d5965-gptlx"
Jul 28 10:40:29.692685 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:29.692665    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef05ab63-fabf-4d18-8221-5f250c8e78e7-proxy-tls\") pod \"isvc-init-fail-505b2f-predictor-5df49d5965-gptlx\" (UID: \"ef05ab63-fabf-4d18-8221-5f250c8e78e7\") " pod="kserve-ci-e2e-test/isvc-init-fail-505b2f-predictor-5df49d5965-gptlx"
Jul 28 10:40:29.697968 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:29.697942    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg2cp\" (UniqueName: \"kubernetes.io/projected/ef05ab63-fabf-4d18-8221-5f250c8e78e7-kube-api-access-mg2cp\") pod \"isvc-init-fail-505b2f-predictor-5df49d5965-gptlx\" (UID: \"ef05ab63-fabf-4d18-8221-5f250c8e78e7\") " pod="kserve-ci-e2e-test/isvc-init-fail-505b2f-predictor-5df49d5965-gptlx"
Jul 28 10:40:29.724131 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:29.724105    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:40:29.865495 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:29.865435    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-505b2f-predictor-5df49d5965-gptlx"
Jul 28 10:40:29.993146 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:29.993000    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-505b2f-predictor-5df49d5965-gptlx"]
Jul 28 10:40:29.995857 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:40:29.995831    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef05ab63_fabf_4d18_8221_5f250c8e78e7.slice/crio-5551b24d7e3bae90242b4503643388aa0c7bf7d086d3df095247c864969ea967 WatchSource:0}: Error finding container 5551b24d7e3bae90242b4503643388aa0c7bf7d086d3df095247c864969ea967: Status 404 returned error can't find the container with id 5551b24d7e3bae90242b4503643388aa0c7bf7d086d3df095247c864969ea967
Jul 28 10:40:30.295773 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:30.295736    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-505b2f-predictor-5df49d5965-gptlx" event={"ID":"ef05ab63-fabf-4d18-8221-5f250c8e78e7","Type":"ContainerStarted","Data":"232b95b40cc0039ec7d361009cd5066abd093e0156a7e42c653ec72bf784c0dd"}
Jul 28 10:40:30.296132 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:30.295804    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-505b2f-predictor-5df49d5965-gptlx" event={"ID":"ef05ab63-fabf-4d18-8221-5f250c8e78e7","Type":"ContainerStarted","Data":"5551b24d7e3bae90242b4503643388aa0c7bf7d086d3df095247c864969ea967"}
Jul 28 10:40:37.329686 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:37.329655    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-505b2f-predictor-5df49d5965-gptlx_ef05ab63-fabf-4d18-8221-5f250c8e78e7/storage-initializer/0.log"
Jul 28 10:40:37.330158 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:37.329695    2583 generic.go:356] "Generic (PLEG): container finished" podID="ef05ab63-fabf-4d18-8221-5f250c8e78e7" containerID="232b95b40cc0039ec7d361009cd5066abd093e0156a7e42c653ec72bf784c0dd" exitCode=1
Jul 28 10:40:37.330158 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:37.329781    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-505b2f-predictor-5df49d5965-gptlx" event={"ID":"ef05ab63-fabf-4d18-8221-5f250c8e78e7","Type":"ContainerDied","Data":"232b95b40cc0039ec7d361009cd5066abd093e0156a7e42c653ec72bf784c0dd"}
Jul 28 10:40:38.336386 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:38.336357    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-505b2f-predictor-5df49d5965-gptlx_ef05ab63-fabf-4d18-8221-5f250c8e78e7/storage-initializer/0.log"
Jul 28 10:40:38.336797 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:38.336412    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-505b2f-predictor-5df49d5965-gptlx" event={"ID":"ef05ab63-fabf-4d18-8221-5f250c8e78e7","Type":"ContainerStarted","Data":"17c3d892eee1dc9d38a1e9da51ffa678dec899f6700a04abffa463aaae04bacf"}
Jul 28 10:40:39.546392 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:39.546363    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-505b2f-predictor-5df49d5965-gptlx"]
Jul 28 10:40:39.546783 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:39.546547    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-505b2f-predictor-5df49d5965-gptlx" podUID="ef05ab63-fabf-4d18-8221-5f250c8e78e7" containerName="storage-initializer" containerID="cri-o://17c3d892eee1dc9d38a1e9da51ffa678dec899f6700a04abffa463aaae04bacf" gracePeriod=30
Jul 28 10:40:39.569910 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:39.569874    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:40:44.091314 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:44.091292    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-505b2f-predictor-5df49d5965-gptlx_ef05ab63-fabf-4d18-8221-5f250c8e78e7/storage-initializer/1.log"
Jul 28 10:40:44.091706 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:44.091690    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-505b2f-predictor-5df49d5965-gptlx_ef05ab63-fabf-4d18-8221-5f250c8e78e7/storage-initializer/0.log"
Jul 28 10:40:44.094034 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:44.094015    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-505b2f-predictor-5df49d5965-gptlx"
Jul 28 10:40:44.192164 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:44.192114    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mg2cp\" (UniqueName: \"kubernetes.io/projected/ef05ab63-fabf-4d18-8221-5f250c8e78e7-kube-api-access-mg2cp\") pod \"ef05ab63-fabf-4d18-8221-5f250c8e78e7\" (UID: \"ef05ab63-fabf-4d18-8221-5f250c8e78e7\") "
Jul 28 10:40:44.192164 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:44.192147    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef05ab63-fabf-4d18-8221-5f250c8e78e7-kube-rbac-proxy-sar-config\") pod \"ef05ab63-fabf-4d18-8221-5f250c8e78e7\" (UID: \"ef05ab63-fabf-4d18-8221-5f250c8e78e7\") "
Jul 28 10:40:44.192356 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:44.192173    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef05ab63-fabf-4d18-8221-5f250c8e78e7-proxy-tls\") pod \"ef05ab63-fabf-4d18-8221-5f250c8e78e7\" (UID: \"ef05ab63-fabf-4d18-8221-5f250c8e78e7\") "
Jul 28 10:40:44.192356 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:44.192251    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef05ab63-fabf-4d18-8221-5f250c8e78e7-kserve-provision-location\") pod \"ef05ab63-fabf-4d18-8221-5f250c8e78e7\" (UID: \"ef05ab63-fabf-4d18-8221-5f250c8e78e7\") "
Jul 28 10:40:44.192356 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:44.192281    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ef05ab63-fabf-4d18-8221-5f250c8e78e7-cabundle-cert\") pod \"ef05ab63-fabf-4d18-8221-5f250c8e78e7\" (UID: \"ef05ab63-fabf-4d18-8221-5f250c8e78e7\") "
Jul 28 10:40:44.192574 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:44.192519    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef05ab63-fabf-4d18-8221-5f250c8e78e7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ef05ab63-fabf-4d18-8221-5f250c8e78e7" (UID: "ef05ab63-fabf-4d18-8221-5f250c8e78e7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:40:44.192693 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:44.192653    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef05ab63-fabf-4d18-8221-5f250c8e78e7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ef05ab63-fabf-4d18-8221-5f250c8e78e7" (UID: "ef05ab63-fabf-4d18-8221-5f250c8e78e7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:40:44.192693 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:44.192679    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef05ab63-fabf-4d18-8221-5f250c8e78e7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ef05ab63-fabf-4d18-8221-5f250c8e78e7" (UID: "ef05ab63-fabf-4d18-8221-5f250c8e78e7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:40:44.194195 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:44.194162    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef05ab63-fabf-4d18-8221-5f250c8e78e7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ef05ab63-fabf-4d18-8221-5f250c8e78e7" (UID: "ef05ab63-fabf-4d18-8221-5f250c8e78e7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:40:44.194295 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:44.194279    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef05ab63-fabf-4d18-8221-5f250c8e78e7-kube-api-access-mg2cp" (OuterVolumeSpecName: "kube-api-access-mg2cp") pod "ef05ab63-fabf-4d18-8221-5f250c8e78e7" (UID: "ef05ab63-fabf-4d18-8221-5f250c8e78e7"). InnerVolumeSpecName "kube-api-access-mg2cp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:40:44.293498 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:44.293473    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef05ab63-fabf-4d18-8221-5f250c8e78e7-kserve-provision-location\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:40:44.293498 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:44.293494    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ef05ab63-fabf-4d18-8221-5f250c8e78e7-cabundle-cert\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:40:44.293617 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:44.293503    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mg2cp\" (UniqueName: \"kubernetes.io/projected/ef05ab63-fabf-4d18-8221-5f250c8e78e7-kube-api-access-mg2cp\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:40:44.293617 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:44.293512    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef05ab63-fabf-4d18-8221-5f250c8e78e7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:40:44.293617 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:44.293522    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef05ab63-fabf-4d18-8221-5f250c8e78e7-proxy-tls\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:40:44.367542 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:44.367518    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-505b2f-predictor-5df49d5965-gptlx_ef05ab63-fabf-4d18-8221-5f250c8e78e7/storage-initializer/1.log"
Jul 28 10:40:44.367919 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:44.367902    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-505b2f-predictor-5df49d5965-gptlx_ef05ab63-fabf-4d18-8221-5f250c8e78e7/storage-initializer/0.log"
Jul 28 10:40:44.368004 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:44.367944    2583 generic.go:356] "Generic (PLEG): container finished" podID="ef05ab63-fabf-4d18-8221-5f250c8e78e7" containerID="17c3d892eee1dc9d38a1e9da51ffa678dec899f6700a04abffa463aaae04bacf" exitCode=1
Jul 28 10:40:44.368004 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:44.367977    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-505b2f-predictor-5df49d5965-gptlx" event={"ID":"ef05ab63-fabf-4d18-8221-5f250c8e78e7","Type":"ContainerDied","Data":"17c3d892eee1dc9d38a1e9da51ffa678dec899f6700a04abffa463aaae04bacf"}
Jul 28 10:40:44.368004 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:44.367994    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-505b2f-predictor-5df49d5965-gptlx"
Jul 28 10:40:44.368159 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:44.368044    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-505b2f-predictor-5df49d5965-gptlx" event={"ID":"ef05ab63-fabf-4d18-8221-5f250c8e78e7","Type":"ContainerDied","Data":"5551b24d7e3bae90242b4503643388aa0c7bf7d086d3df095247c864969ea967"}
Jul 28 10:40:44.368159 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:44.368088    2583 scope.go:117] "RemoveContainer" containerID="17c3d892eee1dc9d38a1e9da51ffa678dec899f6700a04abffa463aaae04bacf"
Jul 28 10:40:44.377279 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:44.377257    2583 scope.go:117] "RemoveContainer" containerID="232b95b40cc0039ec7d361009cd5066abd093e0156a7e42c653ec72bf784c0dd"
Jul 28 10:40:44.384741 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:44.384720    2583 scope.go:117] "RemoveContainer" containerID="17c3d892eee1dc9d38a1e9da51ffa678dec899f6700a04abffa463aaae04bacf"
Jul 28 10:40:44.384990 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:40:44.384973    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17c3d892eee1dc9d38a1e9da51ffa678dec899f6700a04abffa463aaae04bacf\": container with ID starting with 17c3d892eee1dc9d38a1e9da51ffa678dec899f6700a04abffa463aaae04bacf not found: ID does not exist" containerID="17c3d892eee1dc9d38a1e9da51ffa678dec899f6700a04abffa463aaae04bacf"
Jul 28 10:40:44.385047 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:44.384997    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17c3d892eee1dc9d38a1e9da51ffa678dec899f6700a04abffa463aaae04bacf"} err="failed to get container status \"17c3d892eee1dc9d38a1e9da51ffa678dec899f6700a04abffa463aaae04bacf\": rpc error: code = NotFound desc = could not find container \"17c3d892eee1dc9d38a1e9da51ffa678dec899f6700a04abffa463aaae04bacf\": container with ID starting with 17c3d892eee1dc9d38a1e9da51ffa678dec899f6700a04abffa463aaae04bacf not found: ID does not exist"
Jul 28 10:40:44.385047 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:44.385013    2583 scope.go:117] "RemoveContainer" containerID="232b95b40cc0039ec7d361009cd5066abd093e0156a7e42c653ec72bf784c0dd"
Jul 28 10:40:44.385325 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:40:44.385309    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"232b95b40cc0039ec7d361009cd5066abd093e0156a7e42c653ec72bf784c0dd\": container with ID starting with 232b95b40cc0039ec7d361009cd5066abd093e0156a7e42c653ec72bf784c0dd not found: ID does not exist" containerID="232b95b40cc0039ec7d361009cd5066abd093e0156a7e42c653ec72bf784c0dd"
Jul 28 10:40:44.385368 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:44.385329    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"232b95b40cc0039ec7d361009cd5066abd093e0156a7e42c653ec72bf784c0dd"} err="failed to get container status \"232b95b40cc0039ec7d361009cd5066abd093e0156a7e42c653ec72bf784c0dd\": rpc error: code = NotFound desc = could not find container \"232b95b40cc0039ec7d361009cd5066abd093e0156a7e42c653ec72bf784c0dd\": container with ID starting with 232b95b40cc0039ec7d361009cd5066abd093e0156a7e42c653ec72bf784c0dd not found: ID does not exist"
Jul 28 10:40:44.406818 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:44.406794    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-505b2f-predictor-5df49d5965-gptlx"]
Jul 28 10:40:44.414611 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:44.414591    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-505b2f-predictor-5df49d5965-gptlx"]
Jul 28 10:40:44.740447 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:44.740421    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:40:44.778131 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:44.778105    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-1b651-predictor-55d656b769-2v9mr"]
Jul 28 10:40:44.778446 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:44.778430    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef05ab63-fabf-4d18-8221-5f250c8e78e7" containerName="storage-initializer"
Jul 28 10:40:44.778504 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:44.778447    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef05ab63-fabf-4d18-8221-5f250c8e78e7" containerName="storage-initializer"
Jul 28 10:40:44.778558 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:44.778510    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef05ab63-fabf-4d18-8221-5f250c8e78e7" containerName="storage-initializer"
Jul 28 10:40:44.778615 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:44.778565    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef05ab63-fabf-4d18-8221-5f250c8e78e7" containerName="storage-initializer"
Jul 28 10:40:44.778615 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:44.778571    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef05ab63-fabf-4d18-8221-5f250c8e78e7" containerName="storage-initializer"
Jul 28 10:40:44.778720 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:44.778622    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef05ab63-fabf-4d18-8221-5f250c8e78e7" containerName="storage-initializer"
Jul 28 10:40:44.782972 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:44.782955    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-1b651-predictor-55d656b769-2v9mr"
Jul 28 10:40:44.785194 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:44.785159    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 10:40:44.785289 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:44.785203    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 10:40:44.785352 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:44.785290    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 10:40:44.785352 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:44.785294    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-1b651-kube-rbac-proxy-sar-config\""
Jul 28 10:40:44.785585 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:44.785570    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 10:40:44.785661 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:44.785644    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-1b651-predictor-serving-cert\""
Jul 28 10:40:44.785740 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:44.785728    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-d7gr5\""
Jul 28 10:40:44.791466 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:44.791446    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-1b651-predictor-55d656b769-2v9mr"]
Jul 28 10:40:44.895503 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:44.895480    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:40:44.898560 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:44.898545    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f752c\" (UniqueName: \"kubernetes.io/projected/a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4-kube-api-access-f752c\") pod \"raw-sklearn-1b651-predictor-55d656b769-2v9mr\" (UID: \"a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4\") " pod="kserve-ci-e2e-test/raw-sklearn-1b651-predictor-55d656b769-2v9mr"
Jul 28 10:40:44.898633 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:44.898569    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4-kserve-provision-location\") pod \"raw-sklearn-1b651-predictor-55d656b769-2v9mr\" (UID: \"a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4\") " pod="kserve-ci-e2e-test/raw-sklearn-1b651-predictor-55d656b769-2v9mr"
Jul 28 10:40:44.898633 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:44.898606    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4-proxy-tls\") pod \"raw-sklearn-1b651-predictor-55d656b769-2v9mr\" (UID: \"a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4\") " pod="kserve-ci-e2e-test/raw-sklearn-1b651-predictor-55d656b769-2v9mr"
Jul 28 10:40:44.898716 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:44.898672    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-1b651-predictor-55d656b769-2v9mr\" (UID: \"a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4\") " pod="kserve-ci-e2e-test/raw-sklearn-1b651-predictor-55d656b769-2v9mr"
Jul 28 10:40:44.898716 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:44.898697    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4-cabundle-cert\") pod \"raw-sklearn-1b651-predictor-55d656b769-2v9mr\" (UID: \"a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4\") " pod="kserve-ci-e2e-test/raw-sklearn-1b651-predictor-55d656b769-2v9mr"
Jul 28 10:40:45.000067 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:45.000005    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-1b651-predictor-55d656b769-2v9mr\" (UID: \"a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4\") " pod="kserve-ci-e2e-test/raw-sklearn-1b651-predictor-55d656b769-2v9mr"
Jul 28 10:40:45.000067 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:45.000038    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4-cabundle-cert\") pod \"raw-sklearn-1b651-predictor-55d656b769-2v9mr\" (UID: \"a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4\") " pod="kserve-ci-e2e-test/raw-sklearn-1b651-predictor-55d656b769-2v9mr"
Jul 28 10:40:45.000265 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:45.000078    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f752c\" (UniqueName: \"kubernetes.io/projected/a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4-kube-api-access-f752c\") pod \"raw-sklearn-1b651-predictor-55d656b769-2v9mr\" (UID: \"a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4\") " pod="kserve-ci-e2e-test/raw-sklearn-1b651-predictor-55d656b769-2v9mr"
Jul 28 10:40:45.000265 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:45.000096    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4-kserve-provision-location\") pod \"raw-sklearn-1b651-predictor-55d656b769-2v9mr\" (UID: \"a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4\") " pod="kserve-ci-e2e-test/raw-sklearn-1b651-predictor-55d656b769-2v9mr"
Jul 28 10:40:45.000265 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:45.000150    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4-proxy-tls\") pod \"raw-sklearn-1b651-predictor-55d656b769-2v9mr\" (UID: \"a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4\") " pod="kserve-ci-e2e-test/raw-sklearn-1b651-predictor-55d656b769-2v9mr"
Jul 28 10:40:45.000548 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:45.000526    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4-kserve-provision-location\") pod \"raw-sklearn-1b651-predictor-55d656b769-2v9mr\" (UID: \"a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4\") " pod="kserve-ci-e2e-test/raw-sklearn-1b651-predictor-55d656b769-2v9mr"
Jul 28 10:40:45.000732 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:45.000710    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-1b651-predictor-55d656b769-2v9mr\" (UID: \"a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4\") " pod="kserve-ci-e2e-test/raw-sklearn-1b651-predictor-55d656b769-2v9mr"
Jul 28 10:40:45.000795 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:45.000763    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4-cabundle-cert\") pod \"raw-sklearn-1b651-predictor-55d656b769-2v9mr\" (UID: \"a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4\") " pod="kserve-ci-e2e-test/raw-sklearn-1b651-predictor-55d656b769-2v9mr"
Jul 28 10:40:45.002840 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:45.002817    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4-proxy-tls\") pod \"raw-sklearn-1b651-predictor-55d656b769-2v9mr\" (UID: \"a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4\") " pod="kserve-ci-e2e-test/raw-sklearn-1b651-predictor-55d656b769-2v9mr"
Jul 28 10:40:45.008685 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:45.008662    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f752c\" (UniqueName: \"kubernetes.io/projected/a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4-kube-api-access-f752c\") pod \"raw-sklearn-1b651-predictor-55d656b769-2v9mr\" (UID: \"a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4\") " pod="kserve-ci-e2e-test/raw-sklearn-1b651-predictor-55d656b769-2v9mr"
Jul 28 10:40:45.093623 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:45.093592    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-1b651-predictor-55d656b769-2v9mr"
Jul 28 10:40:45.222756 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:45.222723    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-1b651-predictor-55d656b769-2v9mr"]
Jul 28 10:40:45.225433 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:40:45.225393    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda561d8a1_7024_4f6b_a3e5_ab43c9eb51e4.slice/crio-cf78e2411d7493996e3e4c7c5679b3518e211844beeb646de24fe0c9e51eec7d WatchSource:0}: Error finding container cf78e2411d7493996e3e4c7c5679b3518e211844beeb646de24fe0c9e51eec7d: Status 404 returned error can't find the container with id cf78e2411d7493996e3e4c7c5679b3518e211844beeb646de24fe0c9e51eec7d
Jul 28 10:40:45.374424 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:45.374384    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-1b651-predictor-55d656b769-2v9mr" event={"ID":"a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4","Type":"ContainerStarted","Data":"6b0b40104cb1117c6e67ec806115903dcdb0a52628c36004569c895e70521595"}
Jul 28 10:40:45.374573 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:45.374468    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-1b651-predictor-55d656b769-2v9mr" event={"ID":"a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4","Type":"ContainerStarted","Data":"cf78e2411d7493996e3e4c7c5679b3518e211844beeb646de24fe0c9e51eec7d"}
Jul 28 10:40:45.439715 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:45.439685    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef05ab63-fabf-4d18-8221-5f250c8e78e7" path="/var/lib/kubelet/pods/ef05ab63-fabf-4d18-8221-5f250c8e78e7/volumes"
Jul 28 10:40:49.395850 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:49.395817    2583 generic.go:356] "Generic (PLEG): container finished" podID="a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4" containerID="6b0b40104cb1117c6e67ec806115903dcdb0a52628c36004569c895e70521595" exitCode=0
Jul 28 10:40:49.396243 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:49.395891    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-1b651-predictor-55d656b769-2v9mr" event={"ID":"a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4","Type":"ContainerDied","Data":"6b0b40104cb1117c6e67ec806115903dcdb0a52628c36004569c895e70521595"}
Jul 28 10:40:50.404216 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:50.404167    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-1b651-predictor-55d656b769-2v9mr" event={"ID":"a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4","Type":"ContainerStarted","Data":"62873d0ab54ae567ca69881ee3b08f091fc81b63d6ded9d747b4ffc8782f0119"}
Jul 28 10:40:50.404594 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:50.404246    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-1b651-predictor-55d656b769-2v9mr" event={"ID":"a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4","Type":"ContainerStarted","Data":"8326c5b7dedbefbfeb03e1bd7728a33d135dd8531dd5b428f16f71a42d0fdc3f"}
Jul 28 10:40:50.404594 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:50.404432    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-1b651-predictor-55d656b769-2v9mr"
Jul 28 10:40:50.422752 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:50.422706    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-1b651-predictor-55d656b769-2v9mr" podStartSLOduration=2.804562158 podStartE2EDuration="6.422692489s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.618130331s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:40:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:40:50.421305796 +0000 UTC m=+1989.521812349" watchObservedRunningTime="2026-07-28 10:40:50.422692489 +0000 UTC m=+1989.523199044"
Jul 28 10:40:51.408908 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:51.408878    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-1b651-predictor-55d656b769-2v9mr"
Jul 28 10:40:51.410088 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:51.410061    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-1b651-predictor-55d656b769-2v9mr" podUID="a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jul 28 10:40:52.413020 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:52.412982    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-1b651-predictor-55d656b769-2v9mr" podUID="a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jul 28 10:40:57.418983 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:57.418951    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-1b651-predictor-55d656b769-2v9mr"
Jul 28 10:40:57.419460 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:40:57.419431    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-1b651-predictor-55d656b769-2v9mr" podUID="a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jul 28 10:41:07.420148 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:41:07.420107    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-1b651-predictor-55d656b769-2v9mr" podUID="a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jul 28 10:41:17.419826 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:41:17.419784    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-1b651-predictor-55d656b769-2v9mr" podUID="a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jul 28 10:41:27.419939 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:41:27.419902    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-1b651-predictor-55d656b769-2v9mr" podUID="a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jul 28 10:41:34.408314 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:41:34.408276    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:41:37.420341 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:41:37.420297    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-1b651-predictor-55d656b769-2v9mr" podUID="a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jul 28 10:41:47.419421 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:41:47.419381    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-1b651-predictor-55d656b769-2v9mr" podUID="a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jul 28 10:41:57.420852 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:41:57.420822    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-1b651-predictor-55d656b769-2v9mr"
Jul 28 10:42:04.874536 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:04.874454    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-1b651-predictor-55d656b769-2v9mr"]
Jul 28 10:42:04.874969 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:04.874764    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-1b651-predictor-55d656b769-2v9mr" podUID="a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4" containerName="kserve-container" containerID="cri-o://8326c5b7dedbefbfeb03e1bd7728a33d135dd8531dd5b428f16f71a42d0fdc3f" gracePeriod=30
Jul 28 10:42:04.874969 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:04.874793    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-1b651-predictor-55d656b769-2v9mr" podUID="a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4" containerName="kube-rbac-proxy" containerID="cri-o://62873d0ab54ae567ca69881ee3b08f091fc81b63d6ded9d747b4ffc8782f0119" gracePeriod=30
Jul 28 10:42:04.894126 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:04.894083    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:42:05.723676 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:05.723635    2583 generic.go:356] "Generic (PLEG): container finished" podID="a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4" containerID="62873d0ab54ae567ca69881ee3b08f091fc81b63d6ded9d747b4ffc8782f0119" exitCode=2
Jul 28 10:42:05.723845 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:05.723704    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-1b651-predictor-55d656b769-2v9mr" event={"ID":"a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4","Type":"ContainerDied","Data":"62873d0ab54ae567ca69881ee3b08f091fc81b63d6ded9d747b4ffc8782f0119"}
Jul 28 10:42:07.414020 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:07.413973    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-1b651-predictor-55d656b769-2v9mr" podUID="a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.44:8643/healthz\": dial tcp 10.134.0.44:8643: connect: connection refused"
Jul 28 10:42:07.420169 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:07.420136    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-1b651-predictor-55d656b769-2v9mr" podUID="a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jul 28 10:42:09.320161 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:09.320140    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-1b651-predictor-55d656b769-2v9mr"
Jul 28 10:42:09.472185 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:09.472160    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4-cabundle-cert\") pod \"a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4\" (UID: \"a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4\") "
Jul 28 10:42:09.472341 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:09.472232    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4-kserve-provision-location\") pod \"a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4\" (UID: \"a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4\") "
Jul 28 10:42:09.472341 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:09.472272    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4-kube-rbac-proxy-sar-config\") pod \"a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4\" (UID: \"a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4\") "
Jul 28 10:42:09.472341 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:09.472313    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f752c\" (UniqueName: \"kubernetes.io/projected/a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4-kube-api-access-f752c\") pod \"a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4\" (UID: \"a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4\") "
Jul 28 10:42:09.472518 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:09.472481    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4-proxy-tls\") pod \"a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4\" (UID: \"a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4\") "
Jul 28 10:42:09.472597 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:09.472569    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4" (UID: "a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:42:09.472597 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:09.472569    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4" (UID: "a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:42:09.472796 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:09.472778    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4-cabundle-cert\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:42:09.472834 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:09.472799    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4-kserve-provision-location\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:42:09.472834 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:09.472817    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4" (UID: "a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:42:09.474387 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:09.474366    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4-kube-api-access-f752c" (OuterVolumeSpecName: "kube-api-access-f752c") pod "a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4" (UID: "a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4"). InnerVolumeSpecName "kube-api-access-f752c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:42:09.474454 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:09.474441    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4" (UID: "a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:42:09.573592 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:09.573564    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:42:09.573592 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:09.573592    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f752c\" (UniqueName: \"kubernetes.io/projected/a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4-kube-api-access-f752c\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:42:09.573779 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:09.573603    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4-proxy-tls\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:42:09.743992 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:09.743906    2583 generic.go:356] "Generic (PLEG): container finished" podID="a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4" containerID="8326c5b7dedbefbfeb03e1bd7728a33d135dd8531dd5b428f16f71a42d0fdc3f" exitCode=0
Jul 28 10:42:09.743992 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:09.743968    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-1b651-predictor-55d656b769-2v9mr"
Jul 28 10:42:09.744228 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:09.743994    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-1b651-predictor-55d656b769-2v9mr" event={"ID":"a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4","Type":"ContainerDied","Data":"8326c5b7dedbefbfeb03e1bd7728a33d135dd8531dd5b428f16f71a42d0fdc3f"}
Jul 28 10:42:09.744228 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:09.744054    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-1b651-predictor-55d656b769-2v9mr" event={"ID":"a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4","Type":"ContainerDied","Data":"cf78e2411d7493996e3e4c7c5679b3518e211844beeb646de24fe0c9e51eec7d"}
Jul 28 10:42:09.744228 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:09.744080    2583 scope.go:117] "RemoveContainer" containerID="62873d0ab54ae567ca69881ee3b08f091fc81b63d6ded9d747b4ffc8782f0119"
Jul 28 10:42:09.752962 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:09.752942    2583 scope.go:117] "RemoveContainer" containerID="8326c5b7dedbefbfeb03e1bd7728a33d135dd8531dd5b428f16f71a42d0fdc3f"
Jul 28 10:42:09.761325 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:09.761248    2583 scope.go:117] "RemoveContainer" containerID="6b0b40104cb1117c6e67ec806115903dcdb0a52628c36004569c895e70521595"
Jul 28 10:42:09.767654 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:09.767633    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-1b651-predictor-55d656b769-2v9mr"]
Jul 28 10:42:09.768951 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:09.768935    2583 scope.go:117] "RemoveContainer" containerID="62873d0ab54ae567ca69881ee3b08f091fc81b63d6ded9d747b4ffc8782f0119"
Jul 28 10:42:09.769227 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:42:09.769209    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62873d0ab54ae567ca69881ee3b08f091fc81b63d6ded9d747b4ffc8782f0119\": container with ID starting with 62873d0ab54ae567ca69881ee3b08f091fc81b63d6ded9d747b4ffc8782f0119 not found: ID does not exist" containerID="62873d0ab54ae567ca69881ee3b08f091fc81b63d6ded9d747b4ffc8782f0119"
Jul 28 10:42:09.769281 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:09.769236    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62873d0ab54ae567ca69881ee3b08f091fc81b63d6ded9d747b4ffc8782f0119"} err="failed to get container status \"62873d0ab54ae567ca69881ee3b08f091fc81b63d6ded9d747b4ffc8782f0119\": rpc error: code = NotFound desc = could not find container \"62873d0ab54ae567ca69881ee3b08f091fc81b63d6ded9d747b4ffc8782f0119\": container with ID starting with 62873d0ab54ae567ca69881ee3b08f091fc81b63d6ded9d747b4ffc8782f0119 not found: ID does not exist"
Jul 28 10:42:09.769281 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:09.769254    2583 scope.go:117] "RemoveContainer" containerID="8326c5b7dedbefbfeb03e1bd7728a33d135dd8531dd5b428f16f71a42d0fdc3f"
Jul 28 10:42:09.769573 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:42:09.769551    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8326c5b7dedbefbfeb03e1bd7728a33d135dd8531dd5b428f16f71a42d0fdc3f\": container with ID starting with 8326c5b7dedbefbfeb03e1bd7728a33d135dd8531dd5b428f16f71a42d0fdc3f not found: ID does not exist" containerID="8326c5b7dedbefbfeb03e1bd7728a33d135dd8531dd5b428f16f71a42d0fdc3f"
Jul 28 10:42:09.769652 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:09.769580    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8326c5b7dedbefbfeb03e1bd7728a33d135dd8531dd5b428f16f71a42d0fdc3f"} err="failed to get container status \"8326c5b7dedbefbfeb03e1bd7728a33d135dd8531dd5b428f16f71a42d0fdc3f\": rpc error: code = NotFound desc = could not find container \"8326c5b7dedbefbfeb03e1bd7728a33d135dd8531dd5b428f16f71a42d0fdc3f\": container with ID starting with 8326c5b7dedbefbfeb03e1bd7728a33d135dd8531dd5b428f16f71a42d0fdc3f not found: ID does not exist"
Jul 28 10:42:09.769652 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:09.769603    2583 scope.go:117] "RemoveContainer" containerID="6b0b40104cb1117c6e67ec806115903dcdb0a52628c36004569c895e70521595"
Jul 28 10:42:09.769885 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:42:09.769864    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b0b40104cb1117c6e67ec806115903dcdb0a52628c36004569c895e70521595\": container with ID starting with 6b0b40104cb1117c6e67ec806115903dcdb0a52628c36004569c895e70521595 not found: ID does not exist" containerID="6b0b40104cb1117c6e67ec806115903dcdb0a52628c36004569c895e70521595"
Jul 28 10:42:09.769971 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:09.769892    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b0b40104cb1117c6e67ec806115903dcdb0a52628c36004569c895e70521595"} err="failed to get container status \"6b0b40104cb1117c6e67ec806115903dcdb0a52628c36004569c895e70521595\": rpc error: code = NotFound desc = could not find container \"6b0b40104cb1117c6e67ec806115903dcdb0a52628c36004569c895e70521595\": container with ID starting with 6b0b40104cb1117c6e67ec806115903dcdb0a52628c36004569c895e70521595 not found: ID does not exist"
Jul 28 10:42:09.771533 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:09.771513    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-1b651-predictor-55d656b769-2v9mr"]
Jul 28 10:42:10.018045 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:10.017925    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:42:10.040856 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:10.040830    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz"]
Jul 28 10:42:10.041133 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:10.041122    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4" containerName="storage-initializer"
Jul 28 10:42:10.041187 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:10.041135    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4" containerName="storage-initializer"
Jul 28 10:42:10.041187 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:10.041154    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4" containerName="kserve-container"
Jul 28 10:42:10.041187 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:10.041160    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4" containerName="kserve-container"
Jul 28 10:42:10.041187 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:10.041169    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4" containerName="kube-rbac-proxy"
Jul 28 10:42:10.041187 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:10.041174    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4" containerName="kube-rbac-proxy"
Jul 28 10:42:10.041343 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:10.041242    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4" containerName="kserve-container"
Jul 28 10:42:10.041343 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:10.041250    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4" containerName="kube-rbac-proxy"
Jul 28 10:42:10.045659 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:10.045639    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz"
Jul 28 10:42:10.047926 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:10.047903    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 10:42:10.048032 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:10.047912    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-17dc9-kube-rbac-proxy-sar-config\""
Jul 28 10:42:10.048032 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:10.047981    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 10:42:10.048032 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:10.047937    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-17dc9-predictor-serving-cert\""
Jul 28 10:42:10.048314 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:10.048298    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 10:42:10.048696 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:10.048682    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-d7gr5\""
Jul 28 10:42:10.048789 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:10.048763    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 10:42:10.059164 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:10.059145    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz"]
Jul 28 10:42:10.178745 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:10.178716    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fpg4\" (UniqueName: \"kubernetes.io/projected/ea1470b1-b107-47bc-a89e-ef1814214dd2-kube-api-access-8fpg4\") pod \"raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz\" (UID: \"ea1470b1-b107-47bc-a89e-ef1814214dd2\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz"
Jul 28 10:42:10.178745 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:10.178749    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea1470b1-b107-47bc-a89e-ef1814214dd2-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz\" (UID: \"ea1470b1-b107-47bc-a89e-ef1814214dd2\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz"
Jul 28 10:42:10.178924 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:10.178779    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea1470b1-b107-47bc-a89e-ef1814214dd2-kserve-provision-location\") pod \"raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz\" (UID: \"ea1470b1-b107-47bc-a89e-ef1814214dd2\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz"
Jul 28 10:42:10.178924 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:10.178842    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea1470b1-b107-47bc-a89e-ef1814214dd2-cabundle-cert\") pod \"raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz\" (UID: \"ea1470b1-b107-47bc-a89e-ef1814214dd2\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz"
Jul 28 10:42:10.178924 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:10.178879    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea1470b1-b107-47bc-a89e-ef1814214dd2-proxy-tls\") pod \"raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz\" (UID: \"ea1470b1-b107-47bc-a89e-ef1814214dd2\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz"
Jul 28 10:42:10.279977 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:10.279904    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fpg4\" (UniqueName: \"kubernetes.io/projected/ea1470b1-b107-47bc-a89e-ef1814214dd2-kube-api-access-8fpg4\") pod \"raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz\" (UID: \"ea1470b1-b107-47bc-a89e-ef1814214dd2\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz"
Jul 28 10:42:10.279977 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:10.279938    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea1470b1-b107-47bc-a89e-ef1814214dd2-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz\" (UID: \"ea1470b1-b107-47bc-a89e-ef1814214dd2\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz"
Jul 28 10:42:10.279977 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:10.279974    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea1470b1-b107-47bc-a89e-ef1814214dd2-kserve-provision-location\") pod \"raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz\" (UID: \"ea1470b1-b107-47bc-a89e-ef1814214dd2\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz"
Jul 28 10:42:10.280240 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:10.280003    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea1470b1-b107-47bc-a89e-ef1814214dd2-cabundle-cert\") pod \"raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz\" (UID: \"ea1470b1-b107-47bc-a89e-ef1814214dd2\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz"
Jul 28 10:42:10.280240 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:10.280026    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea1470b1-b107-47bc-a89e-ef1814214dd2-proxy-tls\") pod \"raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz\" (UID: \"ea1470b1-b107-47bc-a89e-ef1814214dd2\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz"
Jul 28 10:42:10.280240 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:42:10.280122    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/raw-sklearn-runtime-17dc9-predictor-serving-cert: secret "raw-sklearn-runtime-17dc9-predictor-serving-cert" not found
Jul 28 10:42:10.280240 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:42:10.280202    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea1470b1-b107-47bc-a89e-ef1814214dd2-proxy-tls podName:ea1470b1-b107-47bc-a89e-ef1814214dd2 nodeName:}" failed. No retries permitted until 2026-07-28 10:42:10.780160406 +0000 UTC m=+2069.880666946 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ea1470b1-b107-47bc-a89e-ef1814214dd2-proxy-tls") pod "raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz" (UID: "ea1470b1-b107-47bc-a89e-ef1814214dd2") : secret "raw-sklearn-runtime-17dc9-predictor-serving-cert" not found
Jul 28 10:42:10.280421 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:10.280397    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea1470b1-b107-47bc-a89e-ef1814214dd2-kserve-provision-location\") pod \"raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz\" (UID: \"ea1470b1-b107-47bc-a89e-ef1814214dd2\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz"
Jul 28 10:42:10.280664 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:10.280649    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea1470b1-b107-47bc-a89e-ef1814214dd2-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz\" (UID: \"ea1470b1-b107-47bc-a89e-ef1814214dd2\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz"
Jul 28 10:42:10.280696 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:10.280660    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea1470b1-b107-47bc-a89e-ef1814214dd2-cabundle-cert\") pod \"raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz\" (UID: \"ea1470b1-b107-47bc-a89e-ef1814214dd2\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz"
Jul 28 10:42:10.290610 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:10.290586    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fpg4\" (UniqueName: \"kubernetes.io/projected/ea1470b1-b107-47bc-a89e-ef1814214dd2-kube-api-access-8fpg4\") pod \"raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz\" (UID: \"ea1470b1-b107-47bc-a89e-ef1814214dd2\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz"
Jul 28 10:42:10.389136 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:10.389088    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:42:10.783661 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:10.783617    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea1470b1-b107-47bc-a89e-ef1814214dd2-proxy-tls\") pod \"raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz\" (UID: \"ea1470b1-b107-47bc-a89e-ef1814214dd2\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz"
Jul 28 10:42:10.785967 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:10.785936    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea1470b1-b107-47bc-a89e-ef1814214dd2-proxy-tls\") pod \"raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz\" (UID: \"ea1470b1-b107-47bc-a89e-ef1814214dd2\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz"
Jul 28 10:42:10.955895 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:10.955860    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz"
Jul 28 10:42:11.081567 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:11.081542    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz"]
Jul 28 10:42:11.083153 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:42:11.083127    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea1470b1_b107_47bc_a89e_ef1814214dd2.slice/crio-e77d5889a953eb0e4a818e4af02213c0cee1a74bbb0f35cf8dd483ed1bb8a039 WatchSource:0}: Error finding container e77d5889a953eb0e4a818e4af02213c0cee1a74bbb0f35cf8dd483ed1bb8a039: Status 404 returned error can't find the container with id e77d5889a953eb0e4a818e4af02213c0cee1a74bbb0f35cf8dd483ed1bb8a039
Jul 28 10:42:11.439545 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:11.439505    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4" path="/var/lib/kubelet/pods/a561d8a1-7024-4f6b-a3e5-ab43c9eb51e4/volumes"
Jul 28 10:42:11.756803 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:11.756720    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz" event={"ID":"ea1470b1-b107-47bc-a89e-ef1814214dd2","Type":"ContainerStarted","Data":"b2ea864e2895fdc79572f4e8c1b3b1119b0c9df3422188df76ea99ff921522fd"}
Jul 28 10:42:11.756803 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:11.756789    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz" event={"ID":"ea1470b1-b107-47bc-a89e-ef1814214dd2","Type":"ContainerStarted","Data":"e77d5889a953eb0e4a818e4af02213c0cee1a74bbb0f35cf8dd483ed1bb8a039"}
Jul 28 10:42:15.775088 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:15.775053    2583 generic.go:356] "Generic (PLEG): container finished" podID="ea1470b1-b107-47bc-a89e-ef1814214dd2" containerID="b2ea864e2895fdc79572f4e8c1b3b1119b0c9df3422188df76ea99ff921522fd" exitCode=0
Jul 28 10:42:15.775478 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:15.775103    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz" event={"ID":"ea1470b1-b107-47bc-a89e-ef1814214dd2","Type":"ContainerDied","Data":"b2ea864e2895fdc79572f4e8c1b3b1119b0c9df3422188df76ea99ff921522fd"}
Jul 28 10:42:16.781391 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:16.781359    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz" event={"ID":"ea1470b1-b107-47bc-a89e-ef1814214dd2","Type":"ContainerStarted","Data":"d8f2eaae018330007215c9012dd5344a65b4f5b2c96a4ab1213c009a28c18c6c"}
Jul 28 10:42:16.781762 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:16.781454    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz" event={"ID":"ea1470b1-b107-47bc-a89e-ef1814214dd2","Type":"ContainerStarted","Data":"8ae4151bb7bf8d476c88ad7aec48ad3a4f0019e2b4b04ee88b7fb5fed4a51f81"}
Jul 28 10:42:16.781762 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:16.781647    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz"
Jul 28 10:42:16.800793 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:16.800753    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz" podStartSLOduration=3.03349087 podStartE2EDuration="6.800740842s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.767249972s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:42:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:42:16.800001982 +0000 UTC m=+2075.900508535" watchObservedRunningTime="2026-07-28 10:42:16.800740842 +0000 UTC m=+2075.901247395"
Jul 28 10:42:17.785817 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:17.785781    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz"
Jul 28 10:42:17.786951 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:17.786928    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz" podUID="ea1470b1-b107-47bc-a89e-ef1814214dd2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 28 10:42:18.790202 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:18.790150    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz" podUID="ea1470b1-b107-47bc-a89e-ef1814214dd2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 28 10:42:23.794450 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:23.794422    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz"
Jul 28 10:42:23.794924 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:23.794895    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz" podUID="ea1470b1-b107-47bc-a89e-ef1814214dd2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 28 10:42:33.795236 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:33.795198    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz" podUID="ea1470b1-b107-47bc-a89e-ef1814214dd2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 28 10:42:43.795668 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:43.795621    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz" podUID="ea1470b1-b107-47bc-a89e-ef1814214dd2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 28 10:42:53.795506 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:42:53.795464    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz" podUID="ea1470b1-b107-47bc-a89e-ef1814214dd2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 28 10:43:03.795106 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:43:03.795066    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz" podUID="ea1470b1-b107-47bc-a89e-ef1814214dd2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 28 10:43:13.794974 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:43:13.794925    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz" podUID="ea1470b1-b107-47bc-a89e-ef1814214dd2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 28 10:43:23.795295 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:43:23.795257    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz" podUID="ea1470b1-b107-47bc-a89e-ef1814214dd2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 28 10:43:33.795415 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:43:33.795300    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz"
Jul 28 10:43:40.191869 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:43:40.191831    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz"]
Jul 28 10:43:40.192329 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:43:40.192063    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz" podUID="ea1470b1-b107-47bc-a89e-ef1814214dd2" containerName="kserve-container" containerID="cri-o://8ae4151bb7bf8d476c88ad7aec48ad3a4f0019e2b4b04ee88b7fb5fed4a51f81" gracePeriod=30
Jul 28 10:43:40.192329 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:43:40.192108    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz" podUID="ea1470b1-b107-47bc-a89e-ef1814214dd2" containerName="kube-rbac-proxy" containerID="cri-o://d8f2eaae018330007215c9012dd5344a65b4f5b2c96a4ab1213c009a28c18c6c" gracePeriod=30
Jul 28 10:43:40.217780 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:43:40.217752    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:43:41.147678 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:43:41.147642    2583 generic.go:356] "Generic (PLEG): container finished" podID="ea1470b1-b107-47bc-a89e-ef1814214dd2" containerID="d8f2eaae018330007215c9012dd5344a65b4f5b2c96a4ab1213c009a28c18c6c" exitCode=2
Jul 28 10:43:41.147866 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:43:41.147718    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz" event={"ID":"ea1470b1-b107-47bc-a89e-ef1814214dd2","Type":"ContainerDied","Data":"d8f2eaae018330007215c9012dd5344a65b4f5b2c96a4ab1213c009a28c18c6c"}
Jul 28 10:43:43.791093 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:43:43.791046    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz" podUID="ea1470b1-b107-47bc-a89e-ef1814214dd2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.45:8643/healthz\": dial tcp 10.134.0.45:8643: connect: connection refused"
Jul 28 10:43:43.795416 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:43:43.795390    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz" podUID="ea1470b1-b107-47bc-a89e-ef1814214dd2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 28 10:43:44.646025 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:43:44.646002    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz"
Jul 28 10:43:44.753148 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:43:44.753070    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea1470b1-b107-47bc-a89e-ef1814214dd2-proxy-tls\") pod \"ea1470b1-b107-47bc-a89e-ef1814214dd2\" (UID: \"ea1470b1-b107-47bc-a89e-ef1814214dd2\") "
Jul 28 10:43:44.753148 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:43:44.753128    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea1470b1-b107-47bc-a89e-ef1814214dd2-cabundle-cert\") pod \"ea1470b1-b107-47bc-a89e-ef1814214dd2\" (UID: \"ea1470b1-b107-47bc-a89e-ef1814214dd2\") "
Jul 28 10:43:44.753388 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:43:44.753164    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8fpg4\" (UniqueName: \"kubernetes.io/projected/ea1470b1-b107-47bc-a89e-ef1814214dd2-kube-api-access-8fpg4\") pod \"ea1470b1-b107-47bc-a89e-ef1814214dd2\" (UID: \"ea1470b1-b107-47bc-a89e-ef1814214dd2\") "
Jul 28 10:43:44.753388 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:43:44.753203    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea1470b1-b107-47bc-a89e-ef1814214dd2-kube-rbac-proxy-sar-config\") pod \"ea1470b1-b107-47bc-a89e-ef1814214dd2\" (UID: \"ea1470b1-b107-47bc-a89e-ef1814214dd2\") "
Jul 28 10:43:44.753388 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:43:44.753235    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea1470b1-b107-47bc-a89e-ef1814214dd2-kserve-provision-location\") pod \"ea1470b1-b107-47bc-a89e-ef1814214dd2\" (UID: \"ea1470b1-b107-47bc-a89e-ef1814214dd2\") "
Jul 28 10:43:44.753580 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:43:44.753554    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea1470b1-b107-47bc-a89e-ef1814214dd2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ea1470b1-b107-47bc-a89e-ef1814214dd2" (UID: "ea1470b1-b107-47bc-a89e-ef1814214dd2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:43:44.753640 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:43:44.753609    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea1470b1-b107-47bc-a89e-ef1814214dd2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ea1470b1-b107-47bc-a89e-ef1814214dd2" (UID: "ea1470b1-b107-47bc-a89e-ef1814214dd2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:43:44.753640 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:43:44.753622    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea1470b1-b107-47bc-a89e-ef1814214dd2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ea1470b1-b107-47bc-a89e-ef1814214dd2" (UID: "ea1470b1-b107-47bc-a89e-ef1814214dd2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:43:44.755285 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:43:44.755262    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea1470b1-b107-47bc-a89e-ef1814214dd2-kube-api-access-8fpg4" (OuterVolumeSpecName: "kube-api-access-8fpg4") pod "ea1470b1-b107-47bc-a89e-ef1814214dd2" (UID: "ea1470b1-b107-47bc-a89e-ef1814214dd2"). InnerVolumeSpecName "kube-api-access-8fpg4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:43:44.755386 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:43:44.755297    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea1470b1-b107-47bc-a89e-ef1814214dd2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ea1470b1-b107-47bc-a89e-ef1814214dd2" (UID: "ea1470b1-b107-47bc-a89e-ef1814214dd2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:43:44.854773 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:43:44.854746    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fpg4\" (UniqueName: \"kubernetes.io/projected/ea1470b1-b107-47bc-a89e-ef1814214dd2-kube-api-access-8fpg4\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:43:44.854773 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:43:44.854768    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea1470b1-b107-47bc-a89e-ef1814214dd2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:43:44.855105 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:43:44.854778    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea1470b1-b107-47bc-a89e-ef1814214dd2-kserve-provision-location\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:43:44.855105 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:43:44.854789    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea1470b1-b107-47bc-a89e-ef1814214dd2-proxy-tls\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:43:44.855105 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:43:44.854797    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea1470b1-b107-47bc-a89e-ef1814214dd2-cabundle-cert\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:43:45.166601 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:43:45.166574    2583 generic.go:356] "Generic (PLEG): container finished" podID="ea1470b1-b107-47bc-a89e-ef1814214dd2" containerID="8ae4151bb7bf8d476c88ad7aec48ad3a4f0019e2b4b04ee88b7fb5fed4a51f81" exitCode=0
Jul 28 10:43:45.166748 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:43:45.166635    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz"
Jul 28 10:43:45.166748 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:43:45.166656    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz" event={"ID":"ea1470b1-b107-47bc-a89e-ef1814214dd2","Type":"ContainerDied","Data":"8ae4151bb7bf8d476c88ad7aec48ad3a4f0019e2b4b04ee88b7fb5fed4a51f81"}
Jul 28 10:43:45.166748 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:43:45.166723    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz" event={"ID":"ea1470b1-b107-47bc-a89e-ef1814214dd2","Type":"ContainerDied","Data":"e77d5889a953eb0e4a818e4af02213c0cee1a74bbb0f35cf8dd483ed1bb8a039"}
Jul 28 10:43:45.166855 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:43:45.166761    2583 scope.go:117] "RemoveContainer" containerID="d8f2eaae018330007215c9012dd5344a65b4f5b2c96a4ab1213c009a28c18c6c"
Jul 28 10:43:45.175727 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:43:45.175713    2583 scope.go:117] "RemoveContainer" containerID="8ae4151bb7bf8d476c88ad7aec48ad3a4f0019e2b4b04ee88b7fb5fed4a51f81"
Jul 28 10:43:45.185018 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:43:45.184992    2583 scope.go:117] "RemoveContainer" containerID="b2ea864e2895fdc79572f4e8c1b3b1119b0c9df3422188df76ea99ff921522fd"
Jul 28 10:43:45.192442 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:43:45.192328    2583 scope.go:117] "RemoveContainer" containerID="d8f2eaae018330007215c9012dd5344a65b4f5b2c96a4ab1213c009a28c18c6c"
Jul 28 10:43:45.193290 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:43:45.193259    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8f2eaae018330007215c9012dd5344a65b4f5b2c96a4ab1213c009a28c18c6c\": container with ID starting with d8f2eaae018330007215c9012dd5344a65b4f5b2c96a4ab1213c009a28c18c6c not found: ID does not exist" containerID="d8f2eaae018330007215c9012dd5344a65b4f5b2c96a4ab1213c009a28c18c6c"
Jul 28 10:43:45.193391 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:43:45.193303    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8f2eaae018330007215c9012dd5344a65b4f5b2c96a4ab1213c009a28c18c6c"} err="failed to get container status \"d8f2eaae018330007215c9012dd5344a65b4f5b2c96a4ab1213c009a28c18c6c\": rpc error: code = NotFound desc = could not find container \"d8f2eaae018330007215c9012dd5344a65b4f5b2c96a4ab1213c009a28c18c6c\": container with ID starting with d8f2eaae018330007215c9012dd5344a65b4f5b2c96a4ab1213c009a28c18c6c not found: ID does not exist"
Jul 28 10:43:45.193391 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:43:45.193326    2583 scope.go:117] "RemoveContainer" containerID="8ae4151bb7bf8d476c88ad7aec48ad3a4f0019e2b4b04ee88b7fb5fed4a51f81"
Jul 28 10:43:45.193615 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:43:45.193587    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ae4151bb7bf8d476c88ad7aec48ad3a4f0019e2b4b04ee88b7fb5fed4a51f81\": container with ID starting with 8ae4151bb7bf8d476c88ad7aec48ad3a4f0019e2b4b04ee88b7fb5fed4a51f81 not found: ID does not exist" containerID="8ae4151bb7bf8d476c88ad7aec48ad3a4f0019e2b4b04ee88b7fb5fed4a51f81"
Jul 28 10:43:45.193722 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:43:45.193621    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ae4151bb7bf8d476c88ad7aec48ad3a4f0019e2b4b04ee88b7fb5fed4a51f81"} err="failed to get container status \"8ae4151bb7bf8d476c88ad7aec48ad3a4f0019e2b4b04ee88b7fb5fed4a51f81\": rpc error: code = NotFound desc = could not find container \"8ae4151bb7bf8d476c88ad7aec48ad3a4f0019e2b4b04ee88b7fb5fed4a51f81\": container with ID starting with 8ae4151bb7bf8d476c88ad7aec48ad3a4f0019e2b4b04ee88b7fb5fed4a51f81 not found: ID does not exist"
Jul 28 10:43:45.193722 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:43:45.193642    2583 scope.go:117] "RemoveContainer" containerID="b2ea864e2895fdc79572f4e8c1b3b1119b0c9df3422188df76ea99ff921522fd"
Jul 28 10:43:45.193976 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:43:45.193948    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2ea864e2895fdc79572f4e8c1b3b1119b0c9df3422188df76ea99ff921522fd\": container with ID starting with b2ea864e2895fdc79572f4e8c1b3b1119b0c9df3422188df76ea99ff921522fd not found: ID does not exist" containerID="b2ea864e2895fdc79572f4e8c1b3b1119b0c9df3422188df76ea99ff921522fd"
Jul 28 10:43:45.194077 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:43:45.193982    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2ea864e2895fdc79572f4e8c1b3b1119b0c9df3422188df76ea99ff921522fd"} err="failed to get container status \"b2ea864e2895fdc79572f4e8c1b3b1119b0c9df3422188df76ea99ff921522fd\": rpc error: code = NotFound desc = could not find container \"b2ea864e2895fdc79572f4e8c1b3b1119b0c9df3422188df76ea99ff921522fd\": container with ID starting with b2ea864e2895fdc79572f4e8c1b3b1119b0c9df3422188df76ea99ff921522fd not found: ID does not exist"
Jul 28 10:43:45.194621 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:43:45.194601    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz"]
Jul 28 10:43:45.198235 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:43:45.198216    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-17dc9-predictor-6984f87bcc-tp5dz"]
Jul 28 10:43:45.438599 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:43:45.438523    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea1470b1-b107-47bc-a89e-ef1814214dd2" path="/var/lib/kubelet/pods/ea1470b1-b107-47bc-a89e-ef1814214dd2/volumes"
Jul 28 10:43:46.173318 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:43:46.173287    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-slbvq/must-gather-v9dfp"]
Jul 28 10:43:46.173728 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:43:46.173662    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea1470b1-b107-47bc-a89e-ef1814214dd2" containerName="kserve-container"
Jul 28 10:43:46.173728 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:43:46.173676    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea1470b1-b107-47bc-a89e-ef1814214dd2" containerName="kserve-container"
Jul 28 10:43:46.173728 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:43:46.173693    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea1470b1-b107-47bc-a89e-ef1814214dd2" containerName="storage-initializer"
Jul 28 10:43:46.173728 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:43:46.173699    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea1470b1-b107-47bc-a89e-ef1814214dd2" containerName="storage-initializer"
Jul 28 10:43:46.173728 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:43:46.173708    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea1470b1-b107-47bc-a89e-ef1814214dd2" containerName="kube-rbac-proxy"
Jul 28 10:43:46.173728 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:43:46.173716    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea1470b1-b107-47bc-a89e-ef1814214dd2" containerName="kube-rbac-proxy"
Jul 28 10:43:46.173926 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:43:46.173774    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea1470b1-b107-47bc-a89e-ef1814214dd2" containerName="kube-rbac-proxy"
Jul 28 10:43:46.173926 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:43:46.173782    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea1470b1-b107-47bc-a89e-ef1814214dd2" containerName="kserve-container"
Jul 28 10:43:46.177738 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:43:46.177721    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-slbvq/must-gather-v9dfp"
Jul 28 10:43:46.180001 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:43:46.179977    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-slbvq\"/\"default-dockercfg-p4mqv\""
Jul 28 10:43:46.180126 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:43:46.179977    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-slbvq\"/\"openshift-service-ca.crt\""
Jul 28 10:43:46.180252 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:43:46.180236    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-slbvq\"/\"kube-root-ca.crt\""
Jul 28 10:43:46.183747 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:43:46.183718    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:43:46.184438 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:43:46.184416    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-slbvq/must-gather-v9dfp"]
Jul 28 10:43:46.244443 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:43:46.244417    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:43:46.268597 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:43:46.268563    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvwhb\" (UniqueName: \"kubernetes.io/projected/692f73ff-2660-4a30-a381-d9c16c0d158d-kube-api-access-xvwhb\") pod \"must-gather-v9dfp\" (UID: \"692f73ff-2660-4a30-a381-d9c16c0d158d\") " pod="openshift-must-gather-slbvq/must-gather-v9dfp"
Jul 28 10:43:46.268676 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:43:46.268630    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/692f73ff-2660-4a30-a381-d9c16c0d158d-must-gather-output\") pod \"must-gather-v9dfp\" (UID: \"692f73ff-2660-4a30-a381-d9c16c0d158d\") " pod="openshift-must-gather-slbvq/must-gather-v9dfp"
Jul 28 10:43:46.369557 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:43:46.369530    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/692f73ff-2660-4a30-a381-d9c16c0d158d-must-gather-output\") pod \"must-gather-v9dfp\" (UID: \"692f73ff-2660-4a30-a381-d9c16c0d158d\") " pod="openshift-must-gather-slbvq/must-gather-v9dfp"
Jul 28 10:43:46.369700 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:43:46.369584    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvwhb\" (UniqueName: \"kubernetes.io/projected/692f73ff-2660-4a30-a381-d9c16c0d158d-kube-api-access-xvwhb\") pod \"must-gather-v9dfp\" (UID: \"692f73ff-2660-4a30-a381-d9c16c0d158d\") " pod="openshift-must-gather-slbvq/must-gather-v9dfp"
Jul 28 10:43:46.369894 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:43:46.369872    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/692f73ff-2660-4a30-a381-d9c16c0d158d-must-gather-output\") pod \"must-gather-v9dfp\" (UID: \"692f73ff-2660-4a30-a381-d9c16c0d158d\") " pod="openshift-must-gather-slbvq/must-gather-v9dfp"
Jul 28 10:43:46.377366 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:43:46.377344    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvwhb\" (UniqueName: \"kubernetes.io/projected/692f73ff-2660-4a30-a381-d9c16c0d158d-kube-api-access-xvwhb\") pod \"must-gather-v9dfp\" (UID: \"692f73ff-2660-4a30-a381-d9c16c0d158d\") " pod="openshift-must-gather-slbvq/must-gather-v9dfp"
Jul 28 10:43:46.488106 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:43:46.488037    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-slbvq/must-gather-v9dfp"
Jul 28 10:43:46.616453 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:43:46.616428    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-slbvq/must-gather-v9dfp"]
Jul 28 10:43:46.618800 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:43:46.618759    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod692f73ff_2660_4a30_a381_d9c16c0d158d.slice/crio-0e5c1f4b8a3391159434147be6dcb98d86babd2d9a2bed903d21686695d5d5aa WatchSource:0}: Error finding container 0e5c1f4b8a3391159434147be6dcb98d86babd2d9a2bed903d21686695d5d5aa: Status 404 returned error can't find the container with id 0e5c1f4b8a3391159434147be6dcb98d86babd2d9a2bed903d21686695d5d5aa
Jul 28 10:43:46.620675 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:43:46.620659    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 10:43:47.178941 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:43:47.178904    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-slbvq/must-gather-v9dfp" event={"ID":"692f73ff-2660-4a30-a381-d9c16c0d158d","Type":"ContainerStarted","Data":"0e5c1f4b8a3391159434147be6dcb98d86babd2d9a2bed903d21686695d5d5aa"}
Jul 28 10:43:51.953726 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:43:51.953689    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:43:53.212431 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:43:53.212387    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-slbvq/must-gather-v9dfp" event={"ID":"692f73ff-2660-4a30-a381-d9c16c0d158d","Type":"ContainerStarted","Data":"93400367f5b86093845b5d26d7e8b380038e60fece64610b56119b86ab81ecd4"}
Jul 28 10:43:53.212431 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:43:53.212434    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-slbvq/must-gather-v9dfp" event={"ID":"692f73ff-2660-4a30-a381-d9c16c0d158d","Type":"ContainerStarted","Data":"ee4f7fd1fd2293bdb136172a27cf039d0478bb6503a493d5691be3f9af463466"}
Jul 28 10:43:53.227281 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:43:53.227221    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-slbvq/must-gather-v9dfp" podStartSLOduration=1.243545964 podStartE2EDuration="7.227200582s" totalImagesPullingTime="5.983654618s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:43:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:43:53.226829142 +0000 UTC m=+2172.327335717" watchObservedRunningTime="2026-07-28 10:43:53.227200582 +0000 UTC m=+2172.327707137"
Jul 28 10:44:07.313505 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:07.313473    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:44:07.368585 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:07.368549    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:44:15.322702 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:15.322665    2583 generic.go:356] "Generic (PLEG): container finished" podID="692f73ff-2660-4a30-a381-d9c16c0d158d" containerID="ee4f7fd1fd2293bdb136172a27cf039d0478bb6503a493d5691be3f9af463466" exitCode=0
Jul 28 10:44:15.323284 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:15.322743    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-slbvq/must-gather-v9dfp" event={"ID":"692f73ff-2660-4a30-a381-d9c16c0d158d","Type":"ContainerDied","Data":"ee4f7fd1fd2293bdb136172a27cf039d0478bb6503a493d5691be3f9af463466"}
Jul 28 10:44:15.323284 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:15.323169    2583 scope.go:117] "RemoveContainer" containerID="ee4f7fd1fd2293bdb136172a27cf039d0478bb6503a493d5691be3f9af463466"
Jul 28 10:44:15.766830 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:15.766795    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-slbvq_must-gather-v9dfp_692f73ff-2660-4a30-a381-d9c16c0d158d/gather/0.log"
Jul 28 10:44:16.231347 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:16.231314    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:44:16.402347 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:16.402315    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:44:16.497929 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:16.497825    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:44:18.927914 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:18.927885    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-64pm8_07ad2626-9f01-48ec-adb7-c5f4ab340b7a/global-pull-secret-syncer/0.log"
Jul 28 10:44:19.103640 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:19.103613    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log"
Jul 28 10:44:19.241591 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:19.241509    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-110.ec2.internal_c2a2b445e0df91dcaba3ba620484e779/haproxy/0.log"
Jul 28 10:44:21.209916 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:21.209879    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-slbvq/must-gather-v9dfp"]
Jul 28 10:44:21.210370 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:21.210064    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-slbvq/must-gather-v9dfp" podUID="692f73ff-2660-4a30-a381-d9c16c0d158d" containerName="copy" containerID="cri-o://93400367f5b86093845b5d26d7e8b380038e60fece64610b56119b86ab81ecd4" gracePeriod=2
Jul 28 10:44:21.215357 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:44:21.215313    2583 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"692f73ff-2660-4a30-a381-d9c16c0d158d\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"openshift-must-gather-slbvq\"/\"must-gather-v9dfp\": pods \"must-gather-v9dfp\" not found" pod="openshift-must-gather-slbvq/must-gather-v9dfp"
Jul 28 10:44:21.216305 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:21.216278    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-slbvq/must-gather-v9dfp"]
Jul 28 10:44:21.351457 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:21.351431    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-slbvq_must-gather-v9dfp_692f73ff-2660-4a30-a381-d9c16c0d158d/copy/0.log"
Jul 28 10:44:21.351784 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:21.351757    2583 generic.go:356] "Generic (PLEG): container finished" podID="692f73ff-2660-4a30-a381-d9c16c0d158d" containerID="93400367f5b86093845b5d26d7e8b380038e60fece64610b56119b86ab81ecd4" exitCode=143
Jul 28 10:44:21.436870 ip-10-0-132-110 kubenswrapper[2583]: E0728 10:44:21.436839    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-v9dfp\" is forbidden: User \"system:node:ip-10-0-132-110.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-slbvq\": no relationship found between node 'ip-10-0-132-110.ec2.internal' and this object" podUID="692f73ff-2660-4a30-a381-d9c16c0d158d" pod="openshift-must-gather-slbvq/must-gather-v9dfp"
Jul 28 10:44:21.446849 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:21.446829    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-slbvq_must-gather-v9dfp_692f73ff-2660-4a30-a381-d9c16c0d158d/copy/0.log"
Jul 28 10:44:21.448358 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:21.448343    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-slbvq/must-gather-v9dfp"
Jul 28 10:44:21.464686 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:21.464625    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xvwhb\" (UniqueName: \"kubernetes.io/projected/692f73ff-2660-4a30-a381-d9c16c0d158d-kube-api-access-xvwhb\") pod \"692f73ff-2660-4a30-a381-d9c16c0d158d\" (UID: \"692f73ff-2660-4a30-a381-d9c16c0d158d\") "
Jul 28 10:44:21.464793 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:21.464692    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/692f73ff-2660-4a30-a381-d9c16c0d158d-must-gather-output\") pod \"692f73ff-2660-4a30-a381-d9c16c0d158d\" (UID: \"692f73ff-2660-4a30-a381-d9c16c0d158d\") "
Jul 28 10:44:21.465971 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:21.465942    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/692f73ff-2660-4a30-a381-d9c16c0d158d-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "692f73ff-2660-4a30-a381-d9c16c0d158d" (UID: "692f73ff-2660-4a30-a381-d9c16c0d158d"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:44:21.466703 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:21.466687    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/692f73ff-2660-4a30-a381-d9c16c0d158d-kube-api-access-xvwhb" (OuterVolumeSpecName: "kube-api-access-xvwhb") pod "692f73ff-2660-4a30-a381-d9c16c0d158d" (UID: "692f73ff-2660-4a30-a381-d9c16c0d158d"). InnerVolumeSpecName "kube-api-access-xvwhb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:44:21.565576 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:21.565552    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvwhb\" (UniqueName: \"kubernetes.io/projected/692f73ff-2660-4a30-a381-d9c16c0d158d-kube-api-access-xvwhb\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:44:21.565576 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:21.565573    2583 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/692f73ff-2660-4a30-a381-d9c16c0d158d-must-gather-output\") on node \"ip-10-0-132-110.ec2.internal\" DevicePath \"\""
Jul 28 10:44:21.740401 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:21.740324    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:44:21.797914 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:21.797886    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lqnk6_8490536f-2f40-4389-bf3f-d5cab1ab3a00/konnectivity-agent/0.log" limit=1024
Jul 28 10:44:22.357009 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:22.356981    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-slbvq_must-gather-v9dfp_692f73ff-2660-4a30-a381-d9c16c0d158d/copy/0.log"
Jul 28 10:44:22.357476 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:22.357441    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-slbvq/must-gather-v9dfp"
Jul 28 10:44:22.357540 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:22.357504    2583 scope.go:117] "RemoveContainer" containerID="93400367f5b86093845b5d26d7e8b380038e60fece64610b56119b86ab81ecd4"
Jul 28 10:44:22.365412 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:22.365168    2583 scope.go:117] "RemoveContainer" containerID="ee4f7fd1fd2293bdb136172a27cf039d0478bb6503a493d5691be3f9af463466"
Jul 28 10:44:23.438077 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:23.438043    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="692f73ff-2660-4a30-a381-d9c16c0d158d" path="/var/lib/kubelet/pods/692f73ff-2660-4a30-a381-d9c16c0d158d/volumes"
Jul 28 10:44:23.523736 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:23.523703    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_df33571a-1196-4961-8130-351a73ed1a21/alertmanager/0.log"
Jul 28 10:44:23.550888 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:23.550869    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_df33571a-1196-4961-8130-351a73ed1a21/config-reloader/0.log"
Jul 28 10:44:23.583842 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:23.583818    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_df33571a-1196-4961-8130-351a73ed1a21/kube-rbac-proxy-web/0.log"
Jul 28 10:44:23.604258 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:23.604242    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_df33571a-1196-4961-8130-351a73ed1a21/kube-rbac-proxy/0.log"
Jul 28 10:44:23.623940 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:23.623924    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_df33571a-1196-4961-8130-351a73ed1a21/kube-rbac-proxy-metric/0.log"
Jul 28 10:44:23.649203 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:23.649168    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_df33571a-1196-4961-8130-351a73ed1a21/prom-label-proxy/0.log"
Jul 28 10:44:23.673939 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:23.673919    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_df33571a-1196-4961-8130-351a73ed1a21/init-config-reloader/0.log"
Jul 28 10:44:23.743447 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:23.743385    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-wt5j9_8c69d312-24ea-47e7-8f36-1e6a5362010a/kube-state-metrics/0.log"
Jul 28 10:44:23.762703 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:23.762683    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-wt5j9_8c69d312-24ea-47e7-8f36-1e6a5362010a/kube-rbac-proxy-main/0.log"
Jul 28 10:44:23.784997 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:23.784977    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-wt5j9_8c69d312-24ea-47e7-8f36-1e6a5362010a/kube-rbac-proxy-self/0.log"
Jul 28 10:44:23.816698 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:23.816681    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7475fb8885-f76b9_c57cdfc8-d934-4fed-94bd-d5604c238863/metrics-server/0.log"
Jul 28 10:44:23.848714 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:23.848694    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-kzfgt_24de5934-2a55-43e8-a482-db62cea970b5/monitoring-plugin/0.log"
Jul 28 10:44:23.960157 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:23.960129    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-d8tr2_1b50dc93-37d8-40ad-a0c3-f3c36cc72aca/node-exporter/0.log"
Jul 28 10:44:23.985024 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:23.984998    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-d8tr2_1b50dc93-37d8-40ad-a0c3-f3c36cc72aca/kube-rbac-proxy/0.log"
Jul 28 10:44:24.010327 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:24.010269    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-d8tr2_1b50dc93-37d8-40ad-a0c3-f3c36cc72aca/init-textfile/0.log"
Jul 28 10:44:24.117498 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:24.117473    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-d8vg4_73013d79-268e-42f8-8bbd-9638f7b29d7a/kube-rbac-proxy-main/0.log"
Jul 28 10:44:24.136967 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:24.136944    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-d8vg4_73013d79-268e-42f8-8bbd-9638f7b29d7a/kube-rbac-proxy-self/0.log"
Jul 28 10:44:24.160283 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:24.160264    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-d8vg4_73013d79-268e-42f8-8bbd-9638f7b29d7a/openshift-state-metrics/0.log"
Jul 28 10:44:24.461732 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:24.461702    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-747b8f97c9-bwl55_d82bf008-de56-4acb-8005-267b76eea049/telemeter-client/0.log"
Jul 28 10:44:24.482378 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:24.482351    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-747b8f97c9-bwl55_d82bf008-de56-4acb-8005-267b76eea049/reload/0.log"
Jul 28 10:44:24.514923 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:24.514896    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-747b8f97c9-bwl55_d82bf008-de56-4acb-8005-267b76eea049/kube-rbac-proxy/0.log"
Jul 28 10:44:25.908926 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:25.908897    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-ccv4b_f45e9c0a-7761-4581-8ae8-dd652ec7846f/networking-console-plugin/0.log"
Jul 28 10:44:27.278918 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:27.278881    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-pp4wj/perf-node-gather-daemonset-f7hc4"]
Jul 28 10:44:27.279421 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:27.279352    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="692f73ff-2660-4a30-a381-d9c16c0d158d" containerName="copy"
Jul 28 10:44:27.279421 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:27.279372    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="692f73ff-2660-4a30-a381-d9c16c0d158d" containerName="copy"
Jul 28 10:44:27.279421 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:27.279405    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="692f73ff-2660-4a30-a381-d9c16c0d158d" containerName="gather"
Jul 28 10:44:27.279421 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:27.279413    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="692f73ff-2660-4a30-a381-d9c16c0d158d" containerName="gather"
Jul 28 10:44:27.279635 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:27.279500    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="692f73ff-2660-4a30-a381-d9c16c0d158d" containerName="gather"
Jul 28 10:44:27.279635 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:27.279517    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="692f73ff-2660-4a30-a381-d9c16c0d158d" containerName="copy"
Jul 28 10:44:27.282924 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:27.282902    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pp4wj/perf-node-gather-daemonset-f7hc4"
Jul 28 10:44:27.284763 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:27.284747    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pp4wj\"/\"openshift-service-ca.crt\""
Jul 28 10:44:27.285449 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:27.285434    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pp4wj\"/\"kube-root-ca.crt\""
Jul 28 10:44:27.285513 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:27.285458    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-pp4wj\"/\"default-dockercfg-w6bp4\""
Jul 28 10:44:27.291836 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:27.291815    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pp4wj/perf-node-gather-daemonset-f7hc4"]
Jul 28 10:44:27.314000 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:27.313972    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9162d400-a0d6-45ea-a605-70e4809cbe63-proc\") pod \"perf-node-gather-daemonset-f7hc4\" (UID: \"9162d400-a0d6-45ea-a605-70e4809cbe63\") " pod="openshift-must-gather-pp4wj/perf-node-gather-daemonset-f7hc4"
Jul 28 10:44:27.314102 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:27.314008    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgg55\" (UniqueName: \"kubernetes.io/projected/9162d400-a0d6-45ea-a605-70e4809cbe63-kube-api-access-pgg55\") pod \"perf-node-gather-daemonset-f7hc4\" (UID: \"9162d400-a0d6-45ea-a605-70e4809cbe63\") " pod="openshift-must-gather-pp4wj/perf-node-gather-daemonset-f7hc4"
Jul 28 10:44:27.314102 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:27.314041    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9162d400-a0d6-45ea-a605-70e4809cbe63-lib-modules\") pod \"perf-node-gather-daemonset-f7hc4\" (UID: \"9162d400-a0d6-45ea-a605-70e4809cbe63\") " pod="openshift-must-gather-pp4wj/perf-node-gather-daemonset-f7hc4"
Jul 28 10:44:27.314200 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:27.314108    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9162d400-a0d6-45ea-a605-70e4809cbe63-sys\") pod \"perf-node-gather-daemonset-f7hc4\" (UID: \"9162d400-a0d6-45ea-a605-70e4809cbe63\") " pod="openshift-must-gather-pp4wj/perf-node-gather-daemonset-f7hc4"
Jul 28 10:44:27.314200 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:27.314156    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9162d400-a0d6-45ea-a605-70e4809cbe63-podres\") pod \"perf-node-gather-daemonset-f7hc4\" (UID: \"9162d400-a0d6-45ea-a605-70e4809cbe63\") " pod="openshift-must-gather-pp4wj/perf-node-gather-daemonset-f7hc4"
Jul 28 10:44:27.414998 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:27.414972    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9162d400-a0d6-45ea-a605-70e4809cbe63-podres\") pod \"perf-node-gather-daemonset-f7hc4\" (UID: \"9162d400-a0d6-45ea-a605-70e4809cbe63\") " pod="openshift-must-gather-pp4wj/perf-node-gather-daemonset-f7hc4"
Jul 28 10:44:27.415129 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:27.415024    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9162d400-a0d6-45ea-a605-70e4809cbe63-proc\") pod \"perf-node-gather-daemonset-f7hc4\" (UID: \"9162d400-a0d6-45ea-a605-70e4809cbe63\") " pod="openshift-must-gather-pp4wj/perf-node-gather-daemonset-f7hc4"
Jul 28 10:44:27.415129 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:27.415080    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9162d400-a0d6-45ea-a605-70e4809cbe63-proc\") pod \"perf-node-gather-daemonset-f7hc4\" (UID: \"9162d400-a0d6-45ea-a605-70e4809cbe63\") " pod="openshift-must-gather-pp4wj/perf-node-gather-daemonset-f7hc4"
Jul 28 10:44:27.415129 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:27.415103    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgg55\" (UniqueName: \"kubernetes.io/projected/9162d400-a0d6-45ea-a605-70e4809cbe63-kube-api-access-pgg55\") pod \"perf-node-gather-daemonset-f7hc4\" (UID: \"9162d400-a0d6-45ea-a605-70e4809cbe63\") " pod="openshift-must-gather-pp4wj/perf-node-gather-daemonset-f7hc4"
Jul 28 10:44:27.415129 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:27.415124    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9162d400-a0d6-45ea-a605-70e4809cbe63-lib-modules\") pod \"perf-node-gather-daemonset-f7hc4\" (UID: \"9162d400-a0d6-45ea-a605-70e4809cbe63\") " pod="openshift-must-gather-pp4wj/perf-node-gather-daemonset-f7hc4"
Jul 28 10:44:27.415345 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:27.415140    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9162d400-a0d6-45ea-a605-70e4809cbe63-podres\") pod \"perf-node-gather-daemonset-f7hc4\" (UID: \"9162d400-a0d6-45ea-a605-70e4809cbe63\") " pod="openshift-must-gather-pp4wj/perf-node-gather-daemonset-f7hc4"
Jul 28 10:44:27.415345 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:27.415148    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9162d400-a0d6-45ea-a605-70e4809cbe63-sys\") pod \"perf-node-gather-daemonset-f7hc4\" (UID: \"9162d400-a0d6-45ea-a605-70e4809cbe63\") " pod="openshift-must-gather-pp4wj/perf-node-gather-daemonset-f7hc4"
Jul 28 10:44:27.415345 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:27.415172    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9162d400-a0d6-45ea-a605-70e4809cbe63-sys\") pod \"perf-node-gather-daemonset-f7hc4\" (UID: \"9162d400-a0d6-45ea-a605-70e4809cbe63\") " pod="openshift-must-gather-pp4wj/perf-node-gather-daemonset-f7hc4"
Jul 28 10:44:27.415345 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:27.415315    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9162d400-a0d6-45ea-a605-70e4809cbe63-lib-modules\") pod \"perf-node-gather-daemonset-f7hc4\" (UID: \"9162d400-a0d6-45ea-a605-70e4809cbe63\") " pod="openshift-must-gather-pp4wj/perf-node-gather-daemonset-f7hc4"
Jul 28 10:44:27.422435 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:27.422418    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgg55\" (UniqueName: \"kubernetes.io/projected/9162d400-a0d6-45ea-a605-70e4809cbe63-kube-api-access-pgg55\") pod \"perf-node-gather-daemonset-f7hc4\" (UID: \"9162d400-a0d6-45ea-a605-70e4809cbe63\") " pod="openshift-must-gather-pp4wj/perf-node-gather-daemonset-f7hc4"
Jul 28 10:44:27.593327 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:27.593247    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pp4wj/perf-node-gather-daemonset-f7hc4"
Jul 28 10:44:27.718863 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:27.718838    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pp4wj/perf-node-gather-daemonset-f7hc4"]
Jul 28 10:44:27.721297 ip-10-0-132-110 kubenswrapper[2583]: W0728 10:44:27.721272    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9162d400_a0d6_45ea_a605_70e4809cbe63.slice/crio-0c27d4bcb0f774d0d4fd27e8bec6f93bd0bb876e7c812f83aeff51b2fda31d4d WatchSource:0}: Error finding container 0c27d4bcb0f774d0d4fd27e8bec6f93bd0bb876e7c812f83aeff51b2fda31d4d: Status 404 returned error can't find the container with id 0c27d4bcb0f774d0d4fd27e8bec6f93bd0bb876e7c812f83aeff51b2fda31d4d
Jul 28 10:44:28.363311 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:28.363276    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tmddp_be593a74-ef4a-4604-a64f-5e8ee0722fb7/dns/0.log"
Jul 28 10:44:28.384006 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:28.383979    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tmddp_be593a74-ef4a-4604-a64f-5e8ee0722fb7/kube-rbac-proxy/0.log"
Jul 28 10:44:28.385991 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:28.385962    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pp4wj/perf-node-gather-daemonset-f7hc4" event={"ID":"9162d400-a0d6-45ea-a605-70e4809cbe63","Type":"ContainerStarted","Data":"64b6875893db37b331d975126a91fde4867c6ff8817648defc7612a14dcfd009"}
Jul 28 10:44:28.386110 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:28.386033    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pp4wj/perf-node-gather-daemonset-f7hc4" event={"ID":"9162d400-a0d6-45ea-a605-70e4809cbe63","Type":"ContainerStarted","Data":"0c27d4bcb0f774d0d4fd27e8bec6f93bd0bb876e7c812f83aeff51b2fda31d4d"}
Jul 28 10:44:28.386110 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:28.386059    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-pp4wj/perf-node-gather-daemonset-f7hc4"
Jul 28 10:44:28.401204 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:28.401151    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-pp4wj/perf-node-gather-daemonset-f7hc4" podStartSLOduration=1.4011381489999999 podStartE2EDuration="1.401138149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:44:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:44:28.400117149 +0000 UTC m=+2207.500623702" watchObservedRunningTime="2026-07-28 10:44:28.401138149 +0000 UTC m=+2207.501644727"
Jul 28 10:44:28.405335 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:28.405319    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nxpwb_42229f5e-156f-4abd-a2ee-700c9f340775/dns-node-resolver/0.log"
Jul 28 10:44:28.893741 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:28.893708    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-54f7d57974-x8r6k_0a008076-8474-4d60-a865-f058c917827a/registry/0.log"
Jul 28 10:44:28.949924 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:28.949879    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-v94ww_eeccf40b-0096-40d3-b231-d7ce00688cc9/node-ca/0.log"
Jul 28 10:44:30.043961 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:30.043929    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-zqw4b_eb3e24da-20ea-46a2-8a54-3a518d710941/serve-healthcheck-canary/0.log"
Jul 28 10:44:30.451991 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:30.451964    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-h9wkh_1f82a539-9c16-4566-96ae-7d63e58d46d2/kube-rbac-proxy/0.log"
Jul 28 10:44:30.471920 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:30.471901    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-h9wkh_1f82a539-9c16-4566-96ae-7d63e58d46d2/exporter/0.log"
Jul 28 10:44:30.494072 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:30.494052    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-h9wkh_1f82a539-9c16-4566-96ae-7d63e58d46d2/extractor/0.log"
Jul 28 10:44:32.805016 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:32.804971    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-654c557856-tpx8s_772e05b4-f441-49be-9733-dd14f213ddf6/manager/0.log"
Jul 28 10:44:32.823666 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:32.823647    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-jsdb5_c3a1f9c4-3ef5-4c8e-88db-526ed435b7cf/server/0.log"
Jul 28 10:44:32.971644 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:32.971616    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-lq5rg_b46c2c15-92be-452b-9fd7-0ce038fbc9d9/s3-init/0.log"
Jul 28 10:44:32.988000 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:32.987978    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-lq5rg_b46c2c15-92be-452b-9fd7-0ce038fbc9d9/download-hf-model/0.log"
Jul 28 10:44:34.400030 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:34.400004    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-pp4wj/perf-node-gather-daemonset-f7hc4"
Jul 28 10:44:38.344791 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:38.344759    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-545mm_675b6659-b757-4eca-ba86-28b21af90501/kube-multus-additional-cni-plugins/0.log"
Jul 28 10:44:38.366353 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:38.366325    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-545mm_675b6659-b757-4eca-ba86-28b21af90501/egress-router-binary-copy/0.log"
Jul 28 10:44:38.385731 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:38.385707    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-545mm_675b6659-b757-4eca-ba86-28b21af90501/cni-plugins/0.log"
Jul 28 10:44:38.405002 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:38.404981    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-545mm_675b6659-b757-4eca-ba86-28b21af90501/bond-cni-plugin/0.log"
Jul 28 10:44:38.425452 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:38.425404    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-545mm_675b6659-b757-4eca-ba86-28b21af90501/routeoverride-cni/0.log"
Jul 28 10:44:38.445064 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:38.445036    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-545mm_675b6659-b757-4eca-ba86-28b21af90501/whereabouts-cni-bincopy/0.log"
Jul 28 10:44:38.465286 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:38.465269    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-545mm_675b6659-b757-4eca-ba86-28b21af90501/whereabouts-cni/0.log"
Jul 28 10:44:38.992150 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:38.992073    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jlf2c_ab7f977a-7d91-4d97-85ca-f07a9d373ee8/kube-multus/0.log"
Jul 28 10:44:39.039069 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:39.039035    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-ldq5b_19db4187-c2f2-46d9-bb21-ce1d3c12c422/network-metrics-daemon/0.log"
Jul 28 10:44:39.062234 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:39.062199    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-ldq5b_19db4187-c2f2-46d9-bb21-ce1d3c12c422/kube-rbac-proxy/0.log"
Jul 28 10:44:40.025546 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:40.025502    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rrxnl_ca02c39d-85bd-4cea-ab02-19f67555f81a/ovn-controller/0.log"
Jul 28 10:44:40.042753 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:40.042713    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rrxnl_ca02c39d-85bd-4cea-ab02-19f67555f81a/ovn-acl-logging/0.log"
Jul 28 10:44:40.067599 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:40.067569    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rrxnl_ca02c39d-85bd-4cea-ab02-19f67555f81a/kube-rbac-proxy-node/0.log"
Jul 28 10:44:40.087250 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:40.087217    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rrxnl_ca02c39d-85bd-4cea-ab02-19f67555f81a/kube-rbac-proxy-ovn-metrics/0.log"
Jul 28 10:44:40.107754 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:40.107713    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rrxnl_ca02c39d-85bd-4cea-ab02-19f67555f81a/northd/0.log"
Jul 28 10:44:40.127111 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:40.127071    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rrxnl_ca02c39d-85bd-4cea-ab02-19f67555f81a/nbdb/0.log"
Jul 28 10:44:40.147708 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:40.147679    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rrxnl_ca02c39d-85bd-4cea-ab02-19f67555f81a/sbdb/0.log"
Jul 28 10:44:40.189732 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:40.189702    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rrxnl_ca02c39d-85bd-4cea-ab02-19f67555f81a/ovnkube-controller/0.log"
Jul 28 10:44:41.356583 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:41.356550    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-7dfzm_d329552c-d2a3-4960-9130-bc9e211efc18/network-check-target-container/0.log"
Jul 28 10:44:42.211730 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:42.211698    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-5z6x8_1a872d0f-af6f-4fd4-bf8f-f763ce610da4/iptables-alerter/0.log"
Jul 28 10:44:42.871489 ip-10-0-132-110 kubenswrapper[2583]: I0728 10:44:42.871452    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-vvkrx_17fe592b-e2cc-42ad-8803-0944d20e4dc0/tuned/0.log"