Jul 01 16:56:15.616986 ip-10-0-137-180 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 01 16:56:15.616998 ip-10-0-137-180 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 01 16:56:15.617007 ip-10-0-137-180 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 01 16:56:15.617311 ip-10-0-137-180 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 01 16:56:25.760632 ip-10-0-137-180 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 01 16:56:25.760650 ip-10-0-137-180 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 177249763f4f48e8b3a41065c4520fac --
Jul 01 16:58:00.930143 ip-10-0-137-180 systemd[1]: Starting Kubernetes Kubelet...
Jul 01 16:58:01.376877 ip-10-0-137-180 kubenswrapper[2563]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 01 16:58:01.376877 ip-10-0-137-180 kubenswrapper[2563]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 01 16:58:01.376877 ip-10-0-137-180 kubenswrapper[2563]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 01 16:58:01.376877 ip-10-0-137-180 kubenswrapper[2563]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 01 16:58:01.376877 ip-10-0-137-180 kubenswrapper[2563]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 01 16:58:01.379617 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.379531    2563 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 01 16:58:01.384469 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384454    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 16:58:01.384469 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384471    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 16:58:01.384539 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384475    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 16:58:01.384539 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384478    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 16:58:01.384539 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384481    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 16:58:01.384539 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384484    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 16:58:01.384539 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384486    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 16:58:01.384539 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384489    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 16:58:01.384539 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384492    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 16:58:01.384539 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384496    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 16:58:01.384539 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384498    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 16:58:01.384539 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384501    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 16:58:01.384539 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384503    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 16:58:01.384539 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384506    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 16:58:01.384539 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384508    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 16:58:01.384539 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384511    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 16:58:01.384539 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384514    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 16:58:01.384539 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384516    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 16:58:01.384539 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384518    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 16:58:01.384539 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384521    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 16:58:01.384539 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384524    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 16:58:01.384539 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384526    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 16:58:01.385028 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384529    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 16:58:01.385028 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384531    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 16:58:01.385028 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384534    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 16:58:01.385028 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384536    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 16:58:01.385028 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384539    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 16:58:01.385028 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384541    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 16:58:01.385028 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384544    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 16:58:01.385028 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384546    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 16:58:01.385028 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384549    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 16:58:01.385028 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384552    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 16:58:01.385028 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384554    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 16:58:01.385028 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384559    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 16:58:01.385028 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384562    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 16:58:01.385028 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384566    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 16:58:01.385028 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384568    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 16:58:01.385028 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384571    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 16:58:01.385028 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384577    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 16:58:01.385028 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384580    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 16:58:01.385028 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384583    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 16:58:01.385490 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384586    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 16:58:01.385490 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384588    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 16:58:01.385490 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384591    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 16:58:01.385490 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384593    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 16:58:01.385490 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384595    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 16:58:01.385490 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384598    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 16:58:01.385490 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384600    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 16:58:01.385490 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384602    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 16:58:01.385490 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384604    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 16:58:01.385490 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384607    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 16:58:01.385490 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384609    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 16:58:01.385490 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384611    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 16:58:01.385490 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384614    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 16:58:01.385490 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384616    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 16:58:01.385490 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384619    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 16:58:01.385490 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384622    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 16:58:01.385490 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384624    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 16:58:01.385490 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384627    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 16:58:01.385490 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384629    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 16:58:01.385490 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384631    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 16:58:01.385991 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384634    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 16:58:01.385991 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384636    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 16:58:01.385991 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384639    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 16:58:01.385991 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384641    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 16:58:01.385991 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384643    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 16:58:01.385991 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384646    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 16:58:01.385991 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384648    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 16:58:01.385991 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384653    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 16:58:01.385991 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384656    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 16:58:01.385991 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384659    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 16:58:01.385991 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384662    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 16:58:01.385991 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384665    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 16:58:01.385991 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384669    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 16:58:01.385991 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384672    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 16:58:01.385991 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384675    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 16:58:01.385991 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384678    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 16:58:01.385991 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384681    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 16:58:01.385991 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384684    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 16:58:01.385991 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384687    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 16:58:01.386439 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384690    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 16:58:01.386439 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384692    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 16:58:01.386439 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384695    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 16:58:01.386439 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384697    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 16:58:01.386439 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384699    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 16:58:01.386439 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384701    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 16:58:01.386439 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384704    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 16:58:01.386439 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384708    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 16:58:01.386439 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384710    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 16:58:01.386439 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384713    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 16:58:01.386439 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384716    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 16:58:01.386439 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384718    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 16:58:01.386439 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384721    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 16:58:01.386439 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384723    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 16:58:01.386439 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384726    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 16:58:01.386439 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384729    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 16:58:01.386439 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384731    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 16:58:01.386439 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384734    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 16:58:01.386439 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.384736    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 16:58:01.386439 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385459    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 16:58:01.386972 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385466    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 16:58:01.386972 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385469    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 16:58:01.386972 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385472    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 16:58:01.386972 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385475    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 16:58:01.386972 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385477    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 16:58:01.386972 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385480    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 16:58:01.386972 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385483    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 16:58:01.386972 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385485    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 16:58:01.386972 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385487    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 16:58:01.386972 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385490    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 16:58:01.386972 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385492    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 16:58:01.386972 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385495    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 16:58:01.386972 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385497    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 16:58:01.386972 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385500    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 16:58:01.386972 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385503    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 16:58:01.386972 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385505    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 16:58:01.386972 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385508    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 16:58:01.386972 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385510    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 16:58:01.386972 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385512    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 16:58:01.387438 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385516    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 16:58:01.387438 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385518    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 16:58:01.387438 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385521    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 16:58:01.387438 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385524    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 16:58:01.387438 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385526    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 16:58:01.387438 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385529    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 16:58:01.387438 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385531    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 16:58:01.387438 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385534    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 16:58:01.387438 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385537    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 16:58:01.387438 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385539    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 16:58:01.387438 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385541    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 16:58:01.387438 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385544    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 16:58:01.387438 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385547    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 16:58:01.387438 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385551    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 16:58:01.387438 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385554    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 16:58:01.387438 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385556    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 16:58:01.387438 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385559    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 16:58:01.387438 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385561    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 16:58:01.387438 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385564    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 16:58:01.387438 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385567    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 16:58:01.387936 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385570    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 16:58:01.387936 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385572    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 16:58:01.387936 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385575    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 16:58:01.387936 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385577    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 16:58:01.387936 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385579    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 16:58:01.387936 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385582    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 16:58:01.387936 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385584    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 16:58:01.387936 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385587    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 16:58:01.387936 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385589    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 16:58:01.387936 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385591    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 16:58:01.387936 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385595    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 16:58:01.387936 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385599    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 16:58:01.387936 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385604    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 16:58:01.387936 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385608    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 16:58:01.387936 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385610    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 16:58:01.387936 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385613    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 16:58:01.387936 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385617    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 16:58:01.387936 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385620    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 16:58:01.387936 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385623    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 16:58:01.388414 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385626    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 16:58:01.388414 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385629    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 16:58:01.388414 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385632    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 16:58:01.388414 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385634    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 16:58:01.388414 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385638    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 16:58:01.388414 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385640    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 16:58:01.388414 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385643    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 16:58:01.388414 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385646    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 16:58:01.388414 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385648    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 16:58:01.388414 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385651    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 16:58:01.388414 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385654    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 16:58:01.388414 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385656    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 16:58:01.388414 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385666    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 16:58:01.388414 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385669    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 16:58:01.388414 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385672    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 16:58:01.388414 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385674    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 16:58:01.388414 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385676    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 16:58:01.388414 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385679    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 16:58:01.388414 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385681    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 16:58:01.388414 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385684    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 16:58:01.388926 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385686    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 16:58:01.388926 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385688    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 16:58:01.388926 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385691    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 16:58:01.388926 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385693    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 16:58:01.388926 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385696    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 16:58:01.388926 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385698    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 16:58:01.388926 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385701    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 16:58:01.388926 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385703    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 16:58:01.388926 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385706    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 16:58:01.388926 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385708    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 16:58:01.388926 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385711    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 16:58:01.388926 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385713    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 16:58:01.388926 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385716    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 16:58:01.388926 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385719    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 16:58:01.388926 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385721    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 16:58:01.388926 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385724    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 16:58:01.388926 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385726    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 16:58:01.388926 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385729    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 16:58:01.388926 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385731    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 16:58:01.388926 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.385733    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 16:58:01.389417 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386137    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 16:58:01.389417 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386144    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 16:58:01.389417 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386147    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 16:58:01.389417 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386149    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 16:58:01.389417 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386152    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 16:58:01.389417 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386156    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 16:58:01.389417 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386158    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 16:58:01.389417 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386161    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 16:58:01.389417 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386163    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 16:58:01.389417 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386166    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 16:58:01.389417 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386168    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 16:58:01.389417 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386171    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 16:58:01.389417 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386173    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 16:58:01.389417 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386176    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 16:58:01.389417 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386178    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 16:58:01.389417 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386181    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 16:58:01.389417 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386183    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 16:58:01.389417 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386186    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 16:58:01.389417 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386189    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 16:58:01.389417 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386191    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 16:58:01.389924 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386194    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 16:58:01.389924 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386196    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 16:58:01.389924 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386199    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 16:58:01.389924 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386201    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 16:58:01.389924 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386204    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 16:58:01.389924 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386206    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 16:58:01.389924 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386209    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 16:58:01.389924 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386211    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 16:58:01.389924 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386213    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 16:58:01.389924 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386216    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 16:58:01.389924 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386219    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 16:58:01.389924 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386221    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 16:58:01.389924 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386223    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 16:58:01.389924 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386230    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 16:58:01.389924 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386233    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 16:58:01.389924 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386236    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 16:58:01.389924 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386239    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 16:58:01.389924 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386242    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 16:58:01.389924 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386244    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 16:58:01.389924 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386247    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 16:58:01.390418 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386249    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 16:58:01.390418 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386252    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 16:58:01.390418 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386255    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 16:58:01.390418 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386257    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 16:58:01.390418 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386259    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 16:58:01.390418 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386263    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 16:58:01.390418 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386267    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 16:58:01.390418 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386269    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 16:58:01.390418 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386272    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 16:58:01.390418 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386275    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 16:58:01.390418 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386278    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 16:58:01.390418 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386281    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 16:58:01.390418 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386283    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 16:58:01.390418 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386286    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 16:58:01.390418 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386288    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 16:58:01.390418 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386291    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 16:58:01.390418 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386295    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 16:58:01.390418 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386298    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 16:58:01.390418 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386300    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 16:58:01.390894 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386303    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 16:58:01.390894 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386305    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 16:58:01.390894 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386308    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 16:58:01.390894 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386310    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 16:58:01.390894 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386312    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 16:58:01.390894 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386315    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 16:58:01.390894 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386318    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 16:58:01.390894 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386321    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 16:58:01.390894 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386323    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 16:58:01.390894 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386326    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 16:58:01.390894 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386328    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 16:58:01.390894 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386331    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 16:58:01.390894 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386333    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 16:58:01.390894 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386336    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 16:58:01.390894 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386338    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 16:58:01.390894 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386340    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 16:58:01.390894 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386343    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 16:58:01.390894 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386345    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 16:58:01.390894 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386347    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 16:58:01.390894 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386350    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 16:58:01.391382 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386352    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 16:58:01.391382 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386354    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 16:58:01.391382 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386357    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 16:58:01.391382 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386359    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 16:58:01.391382 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386361    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 16:58:01.391382 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386364    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 16:58:01.391382 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386366    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 16:58:01.391382 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386369    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 16:58:01.391382 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386371    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 16:58:01.391382 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386374    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 16:58:01.391382 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386376    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 16:58:01.391382 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386378    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 16:58:01.391382 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386380    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 16:58:01.391382 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386383    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 16:58:01.391382 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386385    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 16:58:01.391382 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386388    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 16:58:01.391382 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386390    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 16:58:01.391382 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386392    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 16:58:01.391382 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386396    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 16:58:01.391382 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.386398    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 16:58:01.391936 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386465    2563 flags.go:64] FLAG: --address="0.0.0.0"
Jul 01 16:58:01.391936 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386472    2563 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 01 16:58:01.391936 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386478    2563 flags.go:64] FLAG: --anonymous-auth="true"
Jul 01 16:58:01.391936 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386484    2563 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 01 16:58:01.391936 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386491    2563 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 01 16:58:01.391936 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386496    2563 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 01 16:58:01.391936 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386502    2563 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 01 16:58:01.391936 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386509    2563 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 01 16:58:01.391936 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386512    2563 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 01 16:58:01.391936 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386515    2563 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 01 16:58:01.391936 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386518    2563 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 01 16:58:01.391936 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386521    2563 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 01 16:58:01.391936 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386524    2563 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 01 16:58:01.391936 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386526    2563 flags.go:64] FLAG: --cgroup-root=""
Jul 01 16:58:01.391936 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386531    2563 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 01 16:58:01.391936 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386533    2563 flags.go:64] FLAG: --client-ca-file=""
Jul 01 16:58:01.391936 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386536    2563 flags.go:64] FLAG: --cloud-provider="external"
Jul 01 16:58:01.391936 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386539    2563 flags.go:64] FLAG: --cluster-dns="[]"
Jul 01 16:58:01.391936 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386544    2563 flags.go:64] FLAG: --cluster-domain=""
Jul 01 16:58:01.391936 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386547    2563 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 01 16:58:01.391936 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386550    2563 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 01 16:58:01.391936 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386553    2563 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 01 16:58:01.391936 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386556    2563 flags.go:64] FLAG: --container-log-max-files="5"
Jul 01 16:58:01.391936 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386560    2563 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 01 16:58:01.392513 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386562    2563 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 01 16:58:01.392513 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386565    2563 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 01 16:58:01.392513 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386568    2563 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 01 16:58:01.392513 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386571    2563 flags.go:64] FLAG: --contention-profiling="false"
Jul 01 16:58:01.392513 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386574    2563 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 01 16:58:01.392513 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386577    2563 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 01 16:58:01.392513 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386581    2563 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 01 16:58:01.392513 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386583    2563 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 01 16:58:01.392513 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386588    2563 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 01 16:58:01.392513 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386591    2563 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 01 16:58:01.392513 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386594    2563 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 01 16:58:01.392513 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386596    2563 flags.go:64] FLAG: --enable-load-reader="false"
Jul 01 16:58:01.392513 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386599    2563 flags.go:64] FLAG: --enable-server="true"
Jul 01 16:58:01.392513 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386602    2563 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 01 16:58:01.392513 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386607    2563 flags.go:64] FLAG: --event-burst="100"
Jul 01 16:58:01.392513 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386610    2563 flags.go:64] FLAG: --event-qps="50"
Jul 01 16:58:01.392513 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386612    2563 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 01 16:58:01.392513 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386615    2563 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 01 16:58:01.392513 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386618    2563 flags.go:64] FLAG: --eviction-hard=""
Jul 01 16:58:01.392513 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386622    2563 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 01 16:58:01.392513 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386624    2563 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 01 16:58:01.392513 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386627    2563 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 01 16:58:01.392513 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386631    2563 flags.go:64] FLAG: --eviction-soft=""
Jul 01 16:58:01.392513 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386634    2563 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 01 16:58:01.392513 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386637    2563 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 01 16:58:01.393228 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386639    2563 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 01 16:58:01.393228 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386642    2563 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 01 16:58:01.393228 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386645    2563 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 01 16:58:01.393228 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386648    2563 flags.go:64] FLAG: --fail-swap-on="true"
Jul 01 16:58:01.393228 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386650    2563 flags.go:64] FLAG: --feature-gates=""
Jul 01 16:58:01.393228 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386654    2563 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 01 16:58:01.393228 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386657    2563 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 01 16:58:01.393228 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386660    2563 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 01 16:58:01.393228 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386662    2563 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 01 16:58:01.393228 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386665    2563 flags.go:64] FLAG: --healthz-port="10248"
Jul 01 16:58:01.393228 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386668    2563 flags.go:64] FLAG: --help="false"
Jul 01 16:58:01.393228 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386671    2563 flags.go:64] FLAG: --hostname-override="ip-10-0-137-180.ec2.internal"
Jul 01 16:58:01.393228 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386674    2563 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 01 16:58:01.393228 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386681    2563 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 01 16:58:01.393228 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386684    2563 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 01 16:58:01.393228 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386687    2563 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 01 16:58:01.393228 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386690    2563 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 01 16:58:01.393228 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386693    2563 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 01 16:58:01.393228 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386696    2563 flags.go:64] FLAG: --image-service-endpoint=""
Jul 01 16:58:01.393228 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386698    2563 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 01 16:58:01.393228 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386701    2563 flags.go:64] FLAG: --kube-api-burst="100"
Jul 01 16:58:01.393228 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386704    2563 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 01 16:58:01.393228 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386707    2563 flags.go:64] FLAG: --kube-api-qps="50"
Jul 01 16:58:01.393228 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386710    2563 flags.go:64] FLAG: --kube-reserved=""
Jul 01 16:58:01.393826 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386713    2563 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 01 16:58:01.393826 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386715    2563 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 01 16:58:01.393826 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386718    2563 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 01 16:58:01.393826 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386721    2563 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 01 16:58:01.393826 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386724    2563 flags.go:64] FLAG: --lock-file=""
Jul 01 16:58:01.393826 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386727    2563 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 01 16:58:01.393826 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386730    2563 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 01 16:58:01.393826 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386733    2563 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 01 16:58:01.393826 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386738    2563 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 01 16:58:01.393826 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386741    2563 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 01 16:58:01.393826 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386744    2563 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 01 16:58:01.393826 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386746    2563 flags.go:64] FLAG: --logging-format="text"
Jul 01 16:58:01.393826 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386749    2563 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 01 16:58:01.393826 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386752    2563 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 01 16:58:01.393826 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386754    2563 flags.go:64] FLAG: --manifest-url=""
Jul 01 16:58:01.393826 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386757    2563 flags.go:64] FLAG: --manifest-url-header=""
Jul 01 16:58:01.393826 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386761    2563 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 01 16:58:01.393826 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386764    2563 flags.go:64] FLAG: --max-open-files="1000000"
Jul 01 16:58:01.393826 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386768    2563 flags.go:64] FLAG: --max-pods="110"
Jul 01 16:58:01.393826 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386771    2563 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 01 16:58:01.393826 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386774    2563 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 01 16:58:01.393826 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386778    2563 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 01 16:58:01.393826 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386781    2563 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 01 16:58:01.393826 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386784    2563 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 01 16:58:01.393826 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386787    2563 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 01 16:58:01.394438 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386790    2563 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 01 16:58:01.394438 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386796    2563 flags.go:64] FLAG: --node-status-max-images="50"
Jul 01 16:58:01.394438 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386799    2563 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 01 16:58:01.394438 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386802    2563 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 01 16:58:01.394438 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386805    2563 flags.go:64] FLAG: --pod-cidr=""
Jul 01 16:58:01.394438 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386808    2563 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8a2632cd31d78133be7b5fb6999469b2acf9b98d077c7f6ae5d0d94083d76b04"
Jul 01 16:58:01.394438 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386813    2563 flags.go:64] FLAG: --pod-manifest-path=""
Jul 01 16:58:01.394438 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386816    2563 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 01 16:58:01.394438 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386819    2563 flags.go:64] FLAG: --pods-per-core="0"
Jul 01 16:58:01.394438 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386822    2563 flags.go:64] FLAG: --port="10250"
Jul 01 16:58:01.394438 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386825    2563 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 01 16:58:01.394438 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386827    2563 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-07e3be819091e1aff"
Jul 01 16:58:01.394438 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386830    2563 flags.go:64] FLAG: --qos-reserved=""
Jul 01 16:58:01.394438 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386834    2563 flags.go:64] FLAG: --read-only-port="10255"
Jul 01 16:58:01.394438 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386837    2563 flags.go:64] FLAG: --register-node="true"
Jul 01 16:58:01.394438 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386840    2563 flags.go:64] FLAG: --register-with-taints=""
Jul 01 16:58:01.394438 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386843    2563 flags.go:64] FLAG: --registry-burst="10"
Jul 01 16:58:01.394438 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386846    2563 flags.go:64] FLAG: --registry-qps="5"
Jul 01 16:58:01.394438 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386849    2563 flags.go:64] FLAG: --reserved-cpus=""
Jul 01 16:58:01.394438 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386851    2563 flags.go:64] FLAG: --reserved-memory=""
Jul 01 16:58:01.394438 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386855    2563 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 01 16:58:01.394438 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386858    2563 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 01 16:58:01.394438 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386861    2563 flags.go:64] FLAG: --rotate-certificates="false"
Jul 01 16:58:01.394438 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386863    2563 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 01 16:58:01.394438 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386866    2563 flags.go:64] FLAG: --runonce="false"
Jul 01 16:58:01.395064 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386869    2563 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 01 16:58:01.395064 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386872    2563 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 01 16:58:01.395064 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386874    2563 flags.go:64] FLAG: --seccomp-default="false"
Jul 01 16:58:01.395064 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386879    2563 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 01 16:58:01.395064 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386894    2563 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 01 16:58:01.395064 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386897    2563 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 01 16:58:01.395064 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386900    2563 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 01 16:58:01.395064 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386903    2563 flags.go:64] FLAG: --storage-driver-password="root"
Jul 01 16:58:01.395064 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386906    2563 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 01 16:58:01.395064 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386908    2563 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 01 16:58:01.395064 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386911    2563 flags.go:64] FLAG: --storage-driver-user="root"
Jul 01 16:58:01.395064 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386914    2563 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 01 16:58:01.395064 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386917    2563 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 01 16:58:01.395064 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386921    2563 flags.go:64] FLAG: --system-cgroups=""
Jul 01 16:58:01.395064 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386923    2563 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 01 16:58:01.395064 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386929    2563 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 01 16:58:01.395064 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386931    2563 flags.go:64] FLAG: --tls-cert-file=""
Jul 01 16:58:01.395064 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386934    2563 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 01 16:58:01.395064 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386938    2563 flags.go:64] FLAG: --tls-min-version=""
Jul 01 16:58:01.395064 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386941    2563 flags.go:64] FLAG: --tls-private-key-file=""
Jul 01 16:58:01.395064 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386945    2563 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 01 16:58:01.395064 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386948    2563 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 01 16:58:01.395064 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386950    2563 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 01 16:58:01.395064 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386953    2563 flags.go:64] FLAG: --v="2"
Jul 01 16:58:01.395064 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386957    2563 flags.go:64] FLAG: --version="false"
Jul 01 16:58:01.395693 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386961    2563 flags.go:64] FLAG: --vmodule=""
Jul 01 16:58:01.395693 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386965    2563 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 01 16:58:01.395693 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.386968    2563 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 01 16:58:01.395693 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387090    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 16:58:01.395693 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387094    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 16:58:01.395693 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387097    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 16:58:01.395693 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387099    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 16:58:01.395693 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387102    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 16:58:01.395693 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387104    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 16:58:01.395693 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387107    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 16:58:01.395693 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387110    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 16:58:01.395693 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387112    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 16:58:01.395693 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387115    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 16:58:01.395693 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387118    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 16:58:01.395693 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387120    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 16:58:01.395693 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387122    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 16:58:01.395693 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387125    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 16:58:01.395693 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387128    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 16:58:01.395693 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387130    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 16:58:01.395693 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387133    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 16:58:01.396196 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387135    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 16:58:01.396196 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387139    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 16:58:01.396196 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387141    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 16:58:01.396196 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387144    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 16:58:01.396196 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387146    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 16:58:01.396196 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387148    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 16:58:01.396196 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387151    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 16:58:01.396196 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387154    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 16:58:01.396196 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387157    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 16:58:01.396196 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387159    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 16:58:01.396196 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387162    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 16:58:01.396196 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387166    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 16:58:01.396196 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387169    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 16:58:01.396196 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387172    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 16:58:01.396196 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387175    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 16:58:01.396196 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387178    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 16:58:01.396196 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387180    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 16:58:01.396196 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387183    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 16:58:01.396196 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387185    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 16:58:01.396196 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387188    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 16:58:01.396685 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387190    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 16:58:01.396685 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387193    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 16:58:01.396685 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387195    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 16:58:01.396685 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387198    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 16:58:01.396685 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387201    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 16:58:01.396685 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387203    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 16:58:01.396685 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387205    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 16:58:01.396685 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387208    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 16:58:01.396685 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387210    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 16:58:01.396685 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387213    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 16:58:01.396685 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387215    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 16:58:01.396685 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387217    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 16:58:01.396685 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387220    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 16:58:01.396685 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387222    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 16:58:01.396685 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387225    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 16:58:01.396685 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387227    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 16:58:01.396685 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387230    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 16:58:01.396685 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387232    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 16:58:01.396685 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387234    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 16:58:01.396685 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387238    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 16:58:01.397184 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387240    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 16:58:01.397184 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387243    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 16:58:01.397184 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387245    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 16:58:01.397184 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387248    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 16:58:01.397184 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387250    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 16:58:01.397184 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387253    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 16:58:01.397184 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387255    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 16:58:01.397184 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387257    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 16:58:01.397184 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387260    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 16:58:01.397184 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387262    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 16:58:01.397184 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387265    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 16:58:01.397184 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387267    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 16:58:01.397184 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387269    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 16:58:01.397184 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387272    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 16:58:01.397184 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387274    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 16:58:01.397184 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387276    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 16:58:01.397184 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387279    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 16:58:01.397184 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387282    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 16:58:01.397184 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387286    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 16:58:01.397667 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387288    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 16:58:01.397667 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387291    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 16:58:01.397667 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387294    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 16:58:01.397667 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387296    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 16:58:01.397667 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387299    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 16:58:01.397667 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387302    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 16:58:01.397667 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387304    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 16:58:01.397667 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387307    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 16:58:01.397667 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387312    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 16:58:01.397667 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387314    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 16:58:01.397667 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387316    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 16:58:01.397667 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387319    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 16:58:01.397667 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387322    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 16:58:01.397667 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387325    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 16:58:01.397667 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387329    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 16:58:01.397667 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387332    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 16:58:01.397667 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387335    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 16:58:01.397667 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387338    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 16:58:01.397667 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387340    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 16:58:01.397667 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387342    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 16:58:01.398165 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387345    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 16:58:01.398165 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387347    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 16:58:01.398165 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.387350    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 16:58:01.398165 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.388133    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 01 16:58:01.398165 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.395407    2563 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 01 16:58:01.398165 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.395423    2563 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 01 16:58:01.398165 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.395446    2563 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 01 16:58:01.398165 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.395452    2563 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 01 16:58:01.398165 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395506    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 16:58:01.398165 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395511    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 16:58:01.398165 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395514    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 16:58:01.398165 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395517    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 16:58:01.398165 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395520    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 16:58:01.398165 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395523    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 16:58:01.398165 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395525    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 16:58:01.398565 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395528    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 16:58:01.398565 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395530    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 16:58:01.398565 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395533    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 16:58:01.398565 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395536    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 16:58:01.398565 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395539    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 16:58:01.398565 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395541    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 16:58:01.398565 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395544    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 16:58:01.398565 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395546    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 16:58:01.398565 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395549    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 16:58:01.398565 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395551    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 16:58:01.398565 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395554    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 16:58:01.398565 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395556    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 16:58:01.398565 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395559    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 16:58:01.398565 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395561    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 16:58:01.398565 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395564    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 16:58:01.398565 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395567    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 16:58:01.398565 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395569    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 16:58:01.398565 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395571    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 16:58:01.398565 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395574    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 16:58:01.398565 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395577    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 16:58:01.399053 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395579    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 16:58:01.399053 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395582    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 16:58:01.399053 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395586    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 16:58:01.399053 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395588    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 16:58:01.399053 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395591    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 16:58:01.399053 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395594    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 16:58:01.399053 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395596    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 16:58:01.399053 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395599    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 16:58:01.399053 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395601    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 16:58:01.399053 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395603    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 16:58:01.399053 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395606    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 16:58:01.399053 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395608    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 16:58:01.399053 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395611    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 16:58:01.399053 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395613    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 16:58:01.399053 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395616    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 16:58:01.399053 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395618    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 16:58:01.399053 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395621    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 16:58:01.399053 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395624    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 16:58:01.399053 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395626    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 16:58:01.399510 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395629    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 16:58:01.399510 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395631    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 16:58:01.399510 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395633    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 16:58:01.399510 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395636    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 16:58:01.399510 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395638    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 16:58:01.399510 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395641    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 16:58:01.399510 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395643    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 16:58:01.399510 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395646    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 16:58:01.399510 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395648    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 16:58:01.399510 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395651    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 16:58:01.399510 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395653    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 16:58:01.399510 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395656    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 16:58:01.399510 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395658    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 16:58:01.399510 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395661    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 16:58:01.399510 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395663    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 16:58:01.399510 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395667    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 16:58:01.399510 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395670    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 16:58:01.399510 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395672    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 16:58:01.399510 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395675    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 16:58:01.399510 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395677    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 16:58:01.400016 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395680    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 16:58:01.400016 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395682    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 16:58:01.400016 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395685    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 16:58:01.400016 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395687    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 16:58:01.400016 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395689    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 16:58:01.400016 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395691    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 16:58:01.400016 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395695    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 16:58:01.400016 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395699    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 16:58:01.400016 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395702    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 16:58:01.400016 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395705    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 16:58:01.400016 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395708    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 16:58:01.400016 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395711    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 16:58:01.400016 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395713    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 16:58:01.400016 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395717    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 16:58:01.400016 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395721    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 16:58:01.400016 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395724    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 16:58:01.400016 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395727    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 16:58:01.400016 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395730    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 16:58:01.400016 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395732    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 16:58:01.400479 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395735    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 16:58:01.400479 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395737    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 16:58:01.400479 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395739    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 16:58:01.400479 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395743    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 16:58:01.400479 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395746    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 16:58:01.400479 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395749    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 16:58:01.400479 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395752    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 16:58:01.400479 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395754    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 16:58:01.400479 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395757    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 16:58:01.400479 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395760    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 16:58:01.400479 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395763    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 16:58:01.400479 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395766    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 16:58:01.400479 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395768    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 16:58:01.400479 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395770    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 16:58:01.400479 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.395775    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 01 16:58:01.400846 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395909    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 16:58:01.400846 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395915    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 16:58:01.400846 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395918    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 16:58:01.400846 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395920    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 16:58:01.400846 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395923    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 16:58:01.400846 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395926    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 16:58:01.400846 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395928    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 16:58:01.400846 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395931    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 16:58:01.400846 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395934    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 16:58:01.400846 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395936    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 16:58:01.400846 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395939    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 16:58:01.400846 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395941    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 16:58:01.400846 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395944    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 16:58:01.400846 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395946    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 16:58:01.400846 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395949    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 16:58:01.400846 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395952    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 16:58:01.400846 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395954    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 16:58:01.400846 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395957    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 16:58:01.400846 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395959    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 16:58:01.400846 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395961    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 16:58:01.401355 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395964    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 16:58:01.401355 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395966    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 16:58:01.401355 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395969    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 16:58:01.401355 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395971    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 16:58:01.401355 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395974    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 16:58:01.401355 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395976    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 16:58:01.401355 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395978    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 16:58:01.401355 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395981    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 16:58:01.401355 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395984    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 16:58:01.401355 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395986    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 16:58:01.401355 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395989    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 16:58:01.401355 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395991    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 16:58:01.401355 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395994    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 16:58:01.401355 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395996    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 16:58:01.401355 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.395999    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 16:58:01.401355 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.396001    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 16:58:01.401355 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.396003    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 16:58:01.401355 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.396006    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 16:58:01.401355 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.396010    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 16:58:01.401355 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.396013    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 16:58:01.401843 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.396016    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 16:58:01.401843 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.396019    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 16:58:01.401843 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.396021    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 16:58:01.401843 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.396024    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 16:58:01.401843 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.396026    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 16:58:01.401843 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.396029    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 16:58:01.401843 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.396031    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 16:58:01.401843 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.396034    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 16:58:01.401843 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.396036    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 16:58:01.401843 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.396038    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 16:58:01.401843 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.396041    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 16:58:01.401843 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.396061    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 16:58:01.401843 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.396064    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 16:58:01.401843 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.396067    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 16:58:01.401843 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.396069    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 16:58:01.401843 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.396072    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 16:58:01.401843 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.396075    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 16:58:01.401843 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.396077    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 16:58:01.401843 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.396080    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 16:58:01.401843 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.396082    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 16:58:01.402340 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.396084    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 16:58:01.402340 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.396087    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 16:58:01.402340 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.396090    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 16:58:01.402340 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.396092    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 16:58:01.402340 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.396095    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 16:58:01.402340 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.396097    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 16:58:01.402340 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.396100    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 16:58:01.402340 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.396102    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 16:58:01.402340 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.396104    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 16:58:01.402340 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.396107    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 16:58:01.402340 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.396110    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 16:58:01.402340 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.396113    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 16:58:01.402340 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.396116    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 16:58:01.402340 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.396119    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 16:58:01.402340 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.396122    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 16:58:01.402340 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.396124    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 16:58:01.402340 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.396127    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 16:58:01.402340 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.396129    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 16:58:01.402340 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.396132    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 16:58:01.402340 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.396135    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 16:58:01.402820 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.396138    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 16:58:01.402820 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.396140    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 16:58:01.402820 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.396143    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 16:58:01.402820 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.396146    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 16:58:01.402820 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.396149    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 16:58:01.402820 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.396151    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 16:58:01.402820 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.396154    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 16:58:01.402820 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.396157    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 16:58:01.402820 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.396160    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 16:58:01.402820 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.396163    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 16:58:01.402820 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.396167    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 16:58:01.402820 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.396170    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 16:58:01.402820 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.396173    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 16:58:01.402820 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.396175    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 16:58:01.402820 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.396178    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 16:58:01.402820 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.396181    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 16:58:01.402820 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.396184    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 16:58:01.402820 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.396187    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 16:58:01.402820 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:01.396189    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 16:58:01.403283 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.396194    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 01 16:58:01.403283 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.396299    2563 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 01 16:58:01.403283 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.398979    2563 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 01 16:58:01.403283 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.400681    2563 server.go:1019] "Starting client certificate rotation"
Jul 01 16:58:01.403283 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.400780    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 01 16:58:01.403283 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.400851    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 01 16:58:01.404555 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.404526    2563 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 01 16:58:01.408270 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.408250    2563 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 01 16:58:01.422602 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.422585    2563 log.go:25] "Validated CRI v1 runtime API"
Jul 01 16:58:01.425038 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.425024    2563 log.go:25] "Validated CRI v1 image API"
Jul 01 16:58:01.425960 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.425941    2563 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 01 16:58:01.430495 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.430476    2563 fs.go:135] Filesystem UUIDs: map[020b705e-2daa-4e2b-b154-ac14b2ef6c9a:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 bf3d75bf-13dc-4f4c-9828-7660c5b4e78f:/dev/nvme0n1p4]
Jul 01 16:58:01.430552 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.430496    2563 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 01 16:58:01.436546 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.436433    2563 manager.go:216] Machine: {Timestamp:2026-07-01 16:58:01.435046211 +0000 UTC m=+0.375644424 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102574 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec290e31d99542c3773daf07e94f6ca9 SystemUUID:ec290e31-d995-42c3-773d-af07e94f6ca9 BootID:17724976-3f4f-48e8-b3a4-1065c4520fac Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:7f:55:dd:da:c5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:7f:55:dd:da:c5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:d2:88:7e:7e:65:38 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 01 16:58:01.436546 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.436535    2563 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 01 16:58:01.436689 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.436611    2563 manager.go:232] Version: {KernelVersion:5.14.0-570.124.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260623-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 01 16:58:01.438797 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.438780    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 01 16:58:01.440018 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.439992    2563 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 01 16:58:01.440197 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.440025    2563 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-180.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 01 16:58:01.440928 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.440915    2563 topology_manager.go:138] "Creating topology manager with none policy"
Jul 01 16:58:01.440988 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.440934    2563 container_manager_linux.go:309] "Creating device plugin manager"
Jul 01 16:58:01.440988 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.440944    2563 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 01 16:58:01.440988 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.440965    2563 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 01 16:58:01.442175 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.442162    2563 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 01 16:58:01.446866 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.446853    2563 state_mem.go:36] "Initialized new in-memory state store"
Jul 01 16:58:01.447008 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.446997    2563 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 01 16:58:01.447063 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.447037    2563 kubelet.go:486] "Attempting to sync node with API server"
Jul 01 16:58:01.447063 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.447053    2563 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 01 16:58:01.447160 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.447074    2563 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 01 16:58:01.447160 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.447088    2563 kubelet.go:398] "Adding apiserver pod source"
Jul 01 16:58:01.447160 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.447100    2563 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 01 16:58:01.447466 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.447452    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 01 16:58:01.447525 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.447474    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 01 16:58:01.449878 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.449863    2563 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 01 16:58:01.451619 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.451604    2563 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 01 16:58:01.451703 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.451642    2563 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 01 16:58:01.451850 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.451836    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 01 16:58:01.451926 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.451856    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 01 16:58:01.451926 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.451865    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 01 16:58:01.451926 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.451873    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 01 16:58:01.451926 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.451903    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 01 16:58:01.451926 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.451912    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 01 16:58:01.452459 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.452449    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 01 16:58:01.452510 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.452462    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 01 16:58:01.452510 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.452473    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 01 16:58:01.452510 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.452482    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 01 16:58:01.452510 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.452494    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 01 16:58:01.452510 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.452507    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 01 16:58:01.453612 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.453593    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 01 16:58:01.453709 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.453624    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 01 16:58:01.457868 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.457853    2563 server.go:1268] "Started kubelet"
Jul 01 16:58:01.457974 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.457945    2563 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 01 16:58:01.458162 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.458123    2563 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 01 16:58:01.458267 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.458184    2563 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 01 16:58:01.458979 ip-10-0-137-180 systemd[1]: Started Kubernetes Kubelet.
Jul 01 16:58:01.459809 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.459706    2563 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 01 16:58:01.460388 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.460362    2563 server.go:310] "Adding debug handlers to kubelet server"
Jul 01 16:58:01.464165 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:01.464134    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 01 16:58:01.464247 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.464173    2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-180.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 01 16:58:01.464247 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:01.464230    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-180.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 01 16:58:01.465255 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:01.465234    2563 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 01 16:58:01.470302 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:01.469193    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-180.ec2.internal.18be38ea4d6d23cf  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-180.ec2.internal,UID:ip-10-0-137-180.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-180.ec2.internal,},FirstTimestamp:2026-07-01 16:58:01.457796047 +0000 UTC m=+0.398394248,LastTimestamp:2026-07-01 16:58:01.457796047 +0000 UTC m=+0.398394248,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-180.ec2.internal,}"
Jul 01 16:58:01.470591 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.470572    2563 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 01 16:58:01.470591 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.470589    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 01 16:58:01.470796 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:01.470652    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-180.ec2.internal\" not found"
Jul 01 16:58:01.471475 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.471367    2563 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 01 16:58:01.471475 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.471383    2563 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 01 16:58:01.471475 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.471392    2563 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 01 16:58:01.471475 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.471395    2563 factory.go:55] Registering systemd factory
Jul 01 16:58:01.471475 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.471433    2563 factory.go:223] Registration of the systemd container factory successfully
Jul 01 16:58:01.471705 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.471488    2563 reconstruct.go:97] "Volume reconstruction finished"
Jul 01 16:58:01.471705 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.471497    2563 reconciler.go:29] "Reconciler: start to sync state"
Jul 01 16:58:01.471705 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.471641    2563 factory.go:153] Registering CRI-O factory
Jul 01 16:58:01.471705 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.471656    2563 factory.go:223] Registration of the crio container factory successfully
Jul 01 16:58:01.471856 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.471708    2563 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 01 16:58:01.471856 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.471745    2563 factory.go:103] Registering Raw factory
Jul 01 16:58:01.471856 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.471763    2563 manager.go:1196] Started watching for new ooms in manager
Jul 01 16:58:01.472812 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.472796    2563 manager.go:349] Starting recovery of all containers
Jul 01 16:58:01.481070 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:01.481022    2563 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-180.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 01 16:58:01.481145 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:01.481082    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 01 16:58:01.483132 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.483113    2563 manager.go:354] Recovery completed
Jul 01 16:58:01.487150 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.487136    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 16:58:01.492864 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.492846    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-180.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 16:58:01.492945 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.492876    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-180.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 16:58:01.492945 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.492902    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-180.ec2.internal" event="NodeHasSufficientPID"
Jul 01 16:58:01.493396 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.493383    2563 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 01 16:58:01.493396 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.493394    2563 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 01 16:58:01.493465 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.493408    2563 state_mem.go:36] "Initialized new in-memory state store"
Jul 01 16:58:01.493465 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.493428    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 16:58:01.494100 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.494085    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-180.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 16:58:01.494153 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.494115    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-180.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 16:58:01.494153 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.494127    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-180.ec2.internal" event="NodeHasSufficientPID"
Jul 01 16:58:01.495244 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:01.495172    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-180.ec2.internal.18be38ea4f8438bf  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-180.ec2.internal,UID:ip-10-0-137-180.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-180.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-180.ec2.internal,},FirstTimestamp:2026-07-01 16:58:01.492863167 +0000 UTC m=+0.433461368,LastTimestamp:2026-07-01 16:58:01.492863167 +0000 UTC m=+0.433461368,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-180.ec2.internal,}"
Jul 01 16:58:01.495654 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.495634    2563 policy_none.go:49] "None policy: Start"
Jul 01 16:58:01.495710 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.495660    2563 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 01 16:58:01.495710 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.495675    2563 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 01 16:58:01.499517 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.499499    2563 policy_none.go:47] "Start"
Jul 01 16:58:01.506386 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:01.506314    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-180.ec2.internal.18be38ea4f84ba01  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-180.ec2.internal,UID:ip-10-0-137-180.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-137-180.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-137-180.ec2.internal,},FirstTimestamp:2026-07-01 16:58:01.492896257 +0000 UTC m=+0.433494459,LastTimestamp:2026-07-01 16:58:01.492896257 +0000 UTC m=+0.433494459,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-180.ec2.internal,}"
Jul 01 16:58:01.514547 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.514531    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lcf4t"
Jul 01 16:58:01.519715 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:01.519638    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-180.ec2.internal.18be38ea4f84e3e3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-180.ec2.internal,UID:ip-10-0-137-180.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-137-180.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-137-180.ec2.internal,},FirstTimestamp:2026-07-01 16:58:01.492906979 +0000 UTC m=+0.433505182,LastTimestamp:2026-07-01 16:58:01.492906979 +0000 UTC m=+0.433505182,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-180.ec2.internal,}"
Jul 01 16:58:01.524798 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.524780    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lcf4t"
Jul 01 16:58:01.533431 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.533417    2563 manager.go:337] "Starting Device Plugin manager"
Jul 01 16:58:01.556424 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:01.533445    2563 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 01 16:58:01.556424 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.533455    2563 server.go:87] "Starting device plugin registration server"
Jul 01 16:58:01.556424 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.533684    2563 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 01 16:58:01.556424 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.533699    2563 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 01 16:58:01.556424 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.533786    2563 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 01 16:58:01.556424 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.533871    2563 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 01 16:58:01.556424 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.533880    2563 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 01 16:58:01.556424 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:01.533970    2563 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-137-180.ec2.internal.18be38ea4f8438bf\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-180.ec2.internal.18be38ea4f8438bf  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-180.ec2.internal,UID:ip-10-0-137-180.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-180.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-180.ec2.internal,},FirstTimestamp:2026-07-01 16:58:01.492863167 +0000 UTC m=+0.433461368,LastTimestamp:2026-07-01 16:58:01.494100899 +0000 UTC m=+0.434699103,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-180.ec2.internal,}"
Jul 01 16:58:01.556424 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:01.534469    2563 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 01 16:58:01.556424 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:01.534517    2563 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-180.ec2.internal\" not found"
Jul 01 16:58:01.556424 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.545939    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 01 16:58:01.556424 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.547431    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 01 16:58:01.556424 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.547469    2563 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 01 16:58:01.556424 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.547490    2563 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 01 16:58:01.556424 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:01.547521    2563 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 01 16:58:01.556424 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.550031    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 16:58:01.634277 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.634195    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 16:58:01.635355 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.635337    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-180.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 16:58:01.635411 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.635372    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-180.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 16:58:01.635411 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.635386    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-180.ec2.internal" event="NodeHasSufficientPID"
Jul 01 16:58:01.635411 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.635409    2563 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-180.ec2.internal"
Jul 01 16:58:01.647584 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.647561    2563 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-180.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-180.ec2.internal"]
Jul 01 16:58:01.647680 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.647637    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 16:58:01.648652 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.648638    2563 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-180.ec2.internal"
Jul 01 16:58:01.648694 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:01.648659    2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-180.ec2.internal\": node \"ip-10-0-137-180.ec2.internal\" not found"
Jul 01 16:58:01.649070 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.649056    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-180.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 16:58:01.649139 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.649082    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-180.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 16:58:01.649139 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.649096    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-180.ec2.internal" event="NodeHasSufficientPID"
Jul 01 16:58:01.651483 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.651471    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 16:58:01.651594 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.651581    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-180.ec2.internal"
Jul 01 16:58:01.651632 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.651606    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 16:58:01.652162 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.652133    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-180.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 16:58:01.652162 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.652159    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-180.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 16:58:01.652311 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.652172    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-180.ec2.internal" event="NodeHasSufficientPID"
Jul 01 16:58:01.652311 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.652219    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-180.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 16:58:01.652311 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.652243    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-180.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 16:58:01.652311 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.652252    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-180.ec2.internal" event="NodeHasSufficientPID"
Jul 01 16:58:01.654471 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.654455    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-180.ec2.internal"
Jul 01 16:58:01.654560 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.654477    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 16:58:01.655098 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.655080    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-180.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 16:58:01.655188 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.655104    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-180.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 16:58:01.655188 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.655113    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-180.ec2.internal" event="NodeHasSufficientPID"
Jul 01 16:58:01.672155 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.672133    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/83e0fb78b992c4b459a7080a59353cae-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-180.ec2.internal\" (UID: \"83e0fb78b992c4b459a7080a59353cae\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-180.ec2.internal"
Jul 01 16:58:01.672226 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.672164    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/83e0fb78b992c4b459a7080a59353cae-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-180.ec2.internal\" (UID: \"83e0fb78b992c4b459a7080a59353cae\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-180.ec2.internal"
Jul 01 16:58:01.672226 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.672190    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4125034d4754060c768adbf37d115970-config\") pod \"kube-apiserver-proxy-ip-10-0-137-180.ec2.internal\" (UID: \"4125034d4754060c768adbf37d115970\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-180.ec2.internal"
Jul 01 16:58:01.675374 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:01.675359    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-180.ec2.internal\" not found"
Jul 01 16:58:01.680541 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:01.680528    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-180.ec2.internal\" not found" node="ip-10-0-137-180.ec2.internal"
Jul 01 16:58:01.684851 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:01.684837    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-180.ec2.internal\" not found" node="ip-10-0-137-180.ec2.internal"
Jul 01 16:58:01.772459 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.772437    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/83e0fb78b992c4b459a7080a59353cae-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-180.ec2.internal\" (UID: \"83e0fb78b992c4b459a7080a59353cae\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-180.ec2.internal"
Jul 01 16:58:01.772576 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.772464    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/83e0fb78b992c4b459a7080a59353cae-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-180.ec2.internal\" (UID: \"83e0fb78b992c4b459a7080a59353cae\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-180.ec2.internal"
Jul 01 16:58:01.772576 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.772485    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4125034d4754060c768adbf37d115970-config\") pod \"kube-apiserver-proxy-ip-10-0-137-180.ec2.internal\" (UID: \"4125034d4754060c768adbf37d115970\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-180.ec2.internal"
Jul 01 16:58:01.772576 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.772483    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/83e0fb78b992c4b459a7080a59353cae-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-180.ec2.internal\" (UID: \"83e0fb78b992c4b459a7080a59353cae\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-180.ec2.internal"
Jul 01 16:58:01.772576 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.772525    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4125034d4754060c768adbf37d115970-config\") pod \"kube-apiserver-proxy-ip-10-0-137-180.ec2.internal\" (UID: \"4125034d4754060c768adbf37d115970\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-180.ec2.internal"
Jul 01 16:58:01.772576 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.772532    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/83e0fb78b992c4b459a7080a59353cae-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-180.ec2.internal\" (UID: \"83e0fb78b992c4b459a7080a59353cae\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-180.ec2.internal"
Jul 01 16:58:01.775553 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:01.775539    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-180.ec2.internal\" not found"
Jul 01 16:58:01.876061 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:01.876032    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-180.ec2.internal\" not found"
Jul 01 16:58:01.976907 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:01.976807    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-180.ec2.internal\" not found"
Jul 01 16:58:01.984936 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.984918    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-180.ec2.internal"
Jul 01 16:58:01.988094 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:01.988080    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-180.ec2.internal"
Jul 01 16:58:02.077897 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:02.077847    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-180.ec2.internal\" not found"
Jul 01 16:58:02.178409 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:02.178380    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-180.ec2.internal\" not found"
Jul 01 16:58:02.278971 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:02.278871    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-180.ec2.internal\" not found"
Jul 01 16:58:02.379468 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:02.379431    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-180.ec2.internal\" not found"
Jul 01 16:58:02.399628 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:02.399600    2563 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 01 16:58:02.399799 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:02.399779    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 01 16:58:02.479527 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:02.479501    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-180.ec2.internal\" not found"
Jul 01 16:58:02.521412 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:02.521386    2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 16:58:02.532879 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:02.532804    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-06-30 16:53:02 +0000 UTC" deadline="2027-12-11 10:14:30.5359108 +0000 UTC"
Jul 01 16:58:02.532879 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:02.532836    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12665h16m28.003079354s"
Jul 01 16:58:02.580315 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:02.580289    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-180.ec2.internal\" not found"
Jul 01 16:58:02.608025 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:02.607997    2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 16:58:02.680801 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:02.680767    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-180.ec2.internal\" not found"
Jul 01 16:58:02.688955 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:02.688932    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 16:58:02.691946 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:02.691922    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-180.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 16:58:02.692058 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:02.691957    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-180.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 16:58:02.692058 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:02.691968    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-180.ec2.internal" event="NodeHasSufficientPID"
Jul 01 16:58:02.751702 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:02.751663    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4125034d4754060c768adbf37d115970.slice/crio-33f89b9cc4561359c65915326f8bf2432dd5af297ea6e529096e070158b5327b WatchSource:0}: Error finding container 33f89b9cc4561359c65915326f8bf2432dd5af297ea6e529096e070158b5327b: Status 404 returned error can't find the container with id 33f89b9cc4561359c65915326f8bf2432dd5af297ea6e529096e070158b5327b
Jul 01 16:58:02.755918 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:02.755880    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 01 16:58:02.781824 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:02.781790    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-180.ec2.internal\" not found"
Jul 01 16:58:02.820306 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:02.820282    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83e0fb78b992c4b459a7080a59353cae.slice/crio-6a6bdce092f971643549fdc162c4ef79a1dcf51ef50a3bddea1aead1e3b848c2 WatchSource:0}: Error finding container 6a6bdce092f971643549fdc162c4ef79a1dcf51ef50a3bddea1aead1e3b848c2: Status 404 returned error can't find the container with id 6a6bdce092f971643549fdc162c4ef79a1dcf51ef50a3bddea1aead1e3b848c2
Jul 01 16:58:02.876277 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:02.876247    2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 16:58:02.971770 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:02.971734    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-180.ec2.internal"
Jul 01 16:58:02.993216 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:02.993186    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 01 16:58:02.994305 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:02.994290    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-180.ec2.internal"
Jul 01 16:58:03.003121 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:03.003105    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 01 16:58:04.396256 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.396176    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 16:58:04.591608 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.591577    2563 apiserver.go:52] "Watching apiserver"
Jul 01 16:58:04.600026 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.600000    2563 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 01 16:58:04.601358 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.601308    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-txjpp","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-180.ec2.internal","openshift-multus/multus-additional-cni-plugins-gh58z","openshift-multus/multus-st8xw","openshift-multus/network-metrics-daemon-fp8j8","openshift-network-operator/iptables-alerter-fdj7p","openshift-ovn-kubernetes/ovnkube-node-j6kft","kube-system/global-pull-secret-syncer-qp9lx","kube-system/konnectivity-agent-fg79l","kube-system/kube-apiserver-proxy-ip-10-0-137-180.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h79ks","openshift-cluster-node-tuning-operator/tuned-7jfww","openshift-dns/node-resolver-5bjn2","openshift-image-registry/node-ca-bjw8j"]
Jul 01 16:58:04.603899 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.603877    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fp8j8"
Jul 01 16:58:04.604017 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:04.603961    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fp8j8" podUID="899ca2bf-8e98-4def-a2f9-ace75d3170a0"
Jul 01 16:58:04.606185 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.606167    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-j6kft"
Jul 01 16:58:04.610753 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.610732    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 01 16:58:04.610846 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.610801    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fdj7p"
Jul 01 16:58:04.610968 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.610949    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 01 16:58:04.611128 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.611107    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 01 16:58:04.611252 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.611236    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 01 16:58:04.611433 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.610732    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5tgvz\""
Jul 01 16:58:04.611548 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.611498    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-st8xw"
Jul 01 16:58:04.611745 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.611725    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 01 16:58:04.611953 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.611937    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 01 16:58:04.613150 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.613130    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-txjpp"
Jul 01 16:58:04.613289 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:04.613203    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-txjpp" podUID="d4b74cf0-0931-4466-bc62-5b807071e600"
Jul 01 16:58:04.613505 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.613487    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 01 16:58:04.613599 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.613514    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-kh6xp\""
Jul 01 16:58:04.613888 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.613873    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 01 16:58:04.614152 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.614135    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 01 16:58:04.614866 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.614516    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 01 16:58:04.614866 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.614649    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 01 16:58:04.614866 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.614741    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 01 16:58:04.614866 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.614827    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 01 16:58:04.617153 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.616901    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 01 16:58:04.617207 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.617175    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-8mhb6\""
Jul 01 16:58:04.618866 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.618479    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gh58z"
Jul 01 16:58:04.621719 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.621698    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 01 16:58:04.621719 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.621706    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-k769t\""
Jul 01 16:58:04.621719 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.621717    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 01 16:58:04.621987 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.621970    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qp9lx"
Jul 01 16:58:04.622058 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:04.622036    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qp9lx" podUID="ab1a046b-4d82-418b-9c94-3700fa11dd87"
Jul 01 16:58:04.624393 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.624371    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fg79l"
Jul 01 16:58:04.627099 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.626636    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/143b1a41-fbb7-467c-8854-ba67eeb0ea3b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j6kft\" (UID: \"143b1a41-fbb7-467c-8854-ba67eeb0ea3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6kft"
Jul 01 16:58:04.627099 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.626675    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/143b1a41-fbb7-467c-8854-ba67eeb0ea3b-log-socket\") pod \"ovnkube-node-j6kft\" (UID: \"143b1a41-fbb7-467c-8854-ba67eeb0ea3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6kft"
Jul 01 16:58:04.627099 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.626694    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/143b1a41-fbb7-467c-8854-ba67eeb0ea3b-ovnkube-config\") pod \"ovnkube-node-j6kft\" (UID: \"143b1a41-fbb7-467c-8854-ba67eeb0ea3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6kft"
Jul 01 16:58:04.627099 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.626710    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ce4b7fc4-39cb-419a-aae4-30587a901329-system-cni-dir\") pod \"multus-st8xw\" (UID: \"ce4b7fc4-39cb-419a-aae4-30587a901329\") " pod="openshift-multus/multus-st8xw"
Jul 01 16:58:04.627099 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.626724    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ce4b7fc4-39cb-419a-aae4-30587a901329-multus-daemon-config\") pod \"multus-st8xw\" (UID: \"ce4b7fc4-39cb-419a-aae4-30587a901329\") " pod="openshift-multus/multus-st8xw"
Jul 01 16:58:04.627099 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.626740    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/143b1a41-fbb7-467c-8854-ba67eeb0ea3b-systemd-units\") pod \"ovnkube-node-j6kft\" (UID: \"143b1a41-fbb7-467c-8854-ba67eeb0ea3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6kft"
Jul 01 16:58:04.627099 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.626759    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/143b1a41-fbb7-467c-8854-ba67eeb0ea3b-host-run-ovn-kubernetes\") pod \"ovnkube-node-j6kft\" (UID: \"143b1a41-fbb7-467c-8854-ba67eeb0ea3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6kft"
Jul 01 16:58:04.627099 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.626775    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ce4b7fc4-39cb-419a-aae4-30587a901329-os-release\") pod \"multus-st8xw\" (UID: \"ce4b7fc4-39cb-419a-aae4-30587a901329\") " pod="openshift-multus/multus-st8xw"
Jul 01 16:58:04.627099 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.626791    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ce4b7fc4-39cb-419a-aae4-30587a901329-cni-binary-copy\") pod \"multus-st8xw\" (UID: \"ce4b7fc4-39cb-419a-aae4-30587a901329\") " pod="openshift-multus/multus-st8xw"
Jul 01 16:58:04.627099 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.626804    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ce4b7fc4-39cb-419a-aae4-30587a901329-host-run-k8s-cni-cncf-io\") pod \"multus-st8xw\" (UID: \"ce4b7fc4-39cb-419a-aae4-30587a901329\") " pod="openshift-multus/multus-st8xw"
Jul 01 16:58:04.627099 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.626825    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ce4b7fc4-39cb-419a-aae4-30587a901329-host-run-netns\") pod \"multus-st8xw\" (UID: \"ce4b7fc4-39cb-419a-aae4-30587a901329\") " pod="openshift-multus/multus-st8xw"
Jul 01 16:58:04.627099 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.626845    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce4b7fc4-39cb-419a-aae4-30587a901329-host-var-lib-kubelet\") pod \"multus-st8xw\" (UID: \"ce4b7fc4-39cb-419a-aae4-30587a901329\") " pod="openshift-multus/multus-st8xw"
Jul 01 16:58:04.627099 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.626859    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0ab87fa2-edd2-4c8b-a666-1efa2c3bae8e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gh58z\" (UID: \"0ab87fa2-edd2-4c8b-a666-1efa2c3bae8e\") " pod="openshift-multus/multus-additional-cni-plugins-gh58z"
Jul 01 16:58:04.627099 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.626874    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0ab87fa2-edd2-4c8b-a666-1efa2c3bae8e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gh58z\" (UID: \"0ab87fa2-edd2-4c8b-a666-1efa2c3bae8e\") " pod="openshift-multus/multus-additional-cni-plugins-gh58z"
Jul 01 16:58:04.627099 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.626889    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/143b1a41-fbb7-467c-8854-ba67eeb0ea3b-etc-openvswitch\") pod \"ovnkube-node-j6kft\" (UID: \"143b1a41-fbb7-467c-8854-ba67eeb0ea3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6kft"
Jul 01 16:58:04.627099 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.626903    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ce4b7fc4-39cb-419a-aae4-30587a901329-host-var-lib-cni-multus\") pod \"multus-st8xw\" (UID: \"ce4b7fc4-39cb-419a-aae4-30587a901329\") " pod="openshift-multus/multus-st8xw"
Jul 01 16:58:04.628020 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.626923    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0ab87fa2-edd2-4c8b-a666-1efa2c3bae8e-cnibin\") pod \"multus-additional-cni-plugins-gh58z\" (UID: \"0ab87fa2-edd2-4c8b-a666-1efa2c3bae8e\") " pod="openshift-multus/multus-additional-cni-plugins-gh58z"
Jul 01 16:58:04.628020 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.626947    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/143b1a41-fbb7-467c-8854-ba67eeb0ea3b-run-openvswitch\") pod \"ovnkube-node-j6kft\" (UID: \"143b1a41-fbb7-467c-8854-ba67eeb0ea3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6kft"
Jul 01 16:58:04.628020 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.626970    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/143b1a41-fbb7-467c-8854-ba67eeb0ea3b-env-overrides\") pod \"ovnkube-node-j6kft\" (UID: \"143b1a41-fbb7-467c-8854-ba67eeb0ea3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6kft"
Jul 01 16:58:04.628020 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.626998    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/143b1a41-fbb7-467c-8854-ba67eeb0ea3b-ovn-node-metrics-cert\") pod \"ovnkube-node-j6kft\" (UID: \"143b1a41-fbb7-467c-8854-ba67eeb0ea3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6kft"
Jul 01 16:58:04.628020 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.627101    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h79ks"
Jul 01 16:58:04.628020 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.627113    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/143b1a41-fbb7-467c-8854-ba67eeb0ea3b-ovnkube-script-lib\") pod \"ovnkube-node-j6kft\" (UID: \"143b1a41-fbb7-467c-8854-ba67eeb0ea3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6kft"
Jul 01 16:58:04.628020 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.627151    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttgw5\" (UniqueName: \"kubernetes.io/projected/143b1a41-fbb7-467c-8854-ba67eeb0ea3b-kube-api-access-ttgw5\") pod \"ovnkube-node-j6kft\" (UID: \"143b1a41-fbb7-467c-8854-ba67eeb0ea3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6kft"
Jul 01 16:58:04.628020 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.627239    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6f94893d-4fd4-4745-9f71-b89dad4d1582-iptables-alerter-script\") pod \"iptables-alerter-fdj7p\" (UID: \"6f94893d-4fd4-4745-9f71-b89dad4d1582\") " pod="openshift-network-operator/iptables-alerter-fdj7p"
Jul 01 16:58:04.628020 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.627270    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ce4b7fc4-39cb-419a-aae4-30587a901329-multus-socket-dir-parent\") pod \"multus-st8xw\" (UID: \"ce4b7fc4-39cb-419a-aae4-30587a901329\") " pod="openshift-multus/multus-st8xw"
Jul 01 16:58:04.628020 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.627338    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0ab87fa2-edd2-4c8b-a666-1efa2c3bae8e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gh58z\" (UID: \"0ab87fa2-edd2-4c8b-a666-1efa2c3bae8e\") " pod="openshift-multus/multus-additional-cni-plugins-gh58z"
Jul 01 16:58:04.628020 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.627390    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkjkt\" (UniqueName: \"kubernetes.io/projected/0ab87fa2-edd2-4c8b-a666-1efa2c3bae8e-kube-api-access-vkjkt\") pod \"multus-additional-cni-plugins-gh58z\" (UID: \"0ab87fa2-edd2-4c8b-a666-1efa2c3bae8e\") " pod="openshift-multus/multus-additional-cni-plugins-gh58z"
Jul 01 16:58:04.628020 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.627415    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/143b1a41-fbb7-467c-8854-ba67eeb0ea3b-host-run-netns\") pod \"ovnkube-node-j6kft\" (UID: \"143b1a41-fbb7-467c-8854-ba67eeb0ea3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6kft"
Jul 01 16:58:04.628020 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.627469    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/143b1a41-fbb7-467c-8854-ba67eeb0ea3b-run-ovn\") pod \"ovnkube-node-j6kft\" (UID: \"143b1a41-fbb7-467c-8854-ba67eeb0ea3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6kft"
Jul 01 16:58:04.628020 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.627498    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/143b1a41-fbb7-467c-8854-ba67eeb0ea3b-node-log\") pod \"ovnkube-node-j6kft\" (UID: \"143b1a41-fbb7-467c-8854-ba67eeb0ea3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6kft"
Jul 01 16:58:04.628020 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.627523    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/143b1a41-fbb7-467c-8854-ba67eeb0ea3b-host-cni-bin\") pod \"ovnkube-node-j6kft\" (UID: \"143b1a41-fbb7-467c-8854-ba67eeb0ea3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6kft"
Jul 01 16:58:04.628020 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.627549    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnkfn\" (UniqueName: \"kubernetes.io/projected/6f94893d-4fd4-4745-9f71-b89dad4d1582-kube-api-access-pnkfn\") pod \"iptables-alerter-fdj7p\" (UID: \"6f94893d-4fd4-4745-9f71-b89dad4d1582\") " pod="openshift-network-operator/iptables-alerter-fdj7p"
Jul 01 16:58:04.628020 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.627573    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ce4b7fc4-39cb-419a-aae4-30587a901329-multus-cni-dir\") pod \"multus-st8xw\" (UID: \"ce4b7fc4-39cb-419a-aae4-30587a901329\") " pod="openshift-multus/multus-st8xw"
Jul 01 16:58:04.628570 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.627596    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ce4b7fc4-39cb-419a-aae4-30587a901329-host-var-lib-cni-bin\") pod \"multus-st8xw\" (UID: \"ce4b7fc4-39cb-419a-aae4-30587a901329\") " pod="openshift-multus/multus-st8xw"
Jul 01 16:58:04.628570 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.627619    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ce4b7fc4-39cb-419a-aae4-30587a901329-multus-conf-dir\") pod \"multus-st8xw\" (UID: \"ce4b7fc4-39cb-419a-aae4-30587a901329\") " pod="openshift-multus/multus-st8xw"
Jul 01 16:58:04.628570 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.627640    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 01 16:58:04.628570 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.627642    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ce4b7fc4-39cb-419a-aae4-30587a901329-host-run-multus-certs\") pod \"multus-st8xw\" (UID: \"ce4b7fc4-39cb-419a-aae4-30587a901329\") " pod="openshift-multus/multus-st8xw"
Jul 01 16:58:04.628570 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.627647    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 01 16:58:04.628570 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.627661    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-hq4hk\""
Jul 01 16:58:04.628570 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.627666    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkwck\" (UniqueName: \"kubernetes.io/projected/d4b74cf0-0931-4466-bc62-5b807071e600-kube-api-access-bkwck\") pod \"network-check-target-txjpp\" (UID: \"d4b74cf0-0931-4466-bc62-5b807071e600\") " pod="openshift-network-diagnostics/network-check-target-txjpp"
Jul 01 16:58:04.628570 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.627689    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0ab87fa2-edd2-4c8b-a666-1efa2c3bae8e-os-release\") pod \"multus-additional-cni-plugins-gh58z\" (UID: \"0ab87fa2-edd2-4c8b-a666-1efa2c3bae8e\") " pod="openshift-multus/multus-additional-cni-plugins-gh58z"
Jul 01 16:58:04.628570 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.627710    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6f94893d-4fd4-4745-9f71-b89dad4d1582-host-slash\") pod \"iptables-alerter-fdj7p\" (UID: \"6f94893d-4fd4-4745-9f71-b89dad4d1582\") " pod="openshift-network-operator/iptables-alerter-fdj7p"
Jul 01 16:58:04.628570 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.627733    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce4b7fc4-39cb-419a-aae4-30587a901329-etc-kubernetes\") pod \"multus-st8xw\" (UID: \"ce4b7fc4-39cb-419a-aae4-30587a901329\") " pod="openshift-multus/multus-st8xw"
Jul 01 16:58:04.628570 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.627756    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rjkh\" (UniqueName: \"kubernetes.io/projected/ce4b7fc4-39cb-419a-aae4-30587a901329-kube-api-access-5rjkh\") pod \"multus-st8xw\" (UID: \"ce4b7fc4-39cb-419a-aae4-30587a901329\") " pod="openshift-multus/multus-st8xw"
Jul 01 16:58:04.628570 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.627780    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ab87fa2-edd2-4c8b-a666-1efa2c3bae8e-system-cni-dir\") pod \"multus-additional-cni-plugins-gh58z\" (UID: \"0ab87fa2-edd2-4c8b-a666-1efa2c3bae8e\") " pod="openshift-multus/multus-additional-cni-plugins-gh58z"
Jul 01 16:58:04.628570 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.627804    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0ab87fa2-edd2-4c8b-a666-1efa2c3bae8e-cni-binary-copy\") pod \"multus-additional-cni-plugins-gh58z\" (UID: \"0ab87fa2-edd2-4c8b-a666-1efa2c3bae8e\") " pod="openshift-multus/multus-additional-cni-plugins-gh58z"
Jul 01 16:58:04.628570 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.627824    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/143b1a41-fbb7-467c-8854-ba67eeb0ea3b-host-kubelet\") pod \"ovnkube-node-j6kft\" (UID: \"143b1a41-fbb7-467c-8854-ba67eeb0ea3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6kft"
Jul 01 16:58:04.628570 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.627865    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/143b1a41-fbb7-467c-8854-ba67eeb0ea3b-host-slash\") pod \"ovnkube-node-j6kft\" (UID: \"143b1a41-fbb7-467c-8854-ba67eeb0ea3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6kft"
Jul 01 16:58:04.628570 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.627887    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/143b1a41-fbb7-467c-8854-ba67eeb0ea3b-run-systemd\") pod \"ovnkube-node-j6kft\" (UID: \"143b1a41-fbb7-467c-8854-ba67eeb0ea3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6kft"
Jul 01 16:58:04.628570 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.627909    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ce4b7fc4-39cb-419a-aae4-30587a901329-cnibin\") pod \"multus-st8xw\" (UID: \"ce4b7fc4-39cb-419a-aae4-30587a901329\") " pod="openshift-multus/multus-st8xw"
Jul 01 16:58:04.628570 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.627940    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ce4b7fc4-39cb-419a-aae4-30587a901329-hostroot\") pod \"multus-st8xw\" (UID: \"ce4b7fc4-39cb-419a-aae4-30587a901329\") " pod="openshift-multus/multus-st8xw"
Jul 01 16:58:04.629337 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.627968    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/899ca2bf-8e98-4def-a2f9-ace75d3170a0-metrics-certs\") pod \"network-metrics-daemon-fp8j8\" (UID: \"899ca2bf-8e98-4def-a2f9-ace75d3170a0\") " pod="openshift-multus/network-metrics-daemon-fp8j8"
Jul 01 16:58:04.629337 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.628004    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gv8z\" (UniqueName: \"kubernetes.io/projected/899ca2bf-8e98-4def-a2f9-ace75d3170a0-kube-api-access-7gv8z\") pod \"network-metrics-daemon-fp8j8\" (UID: \"899ca2bf-8e98-4def-a2f9-ace75d3170a0\") " pod="openshift-multus/network-metrics-daemon-fp8j8"
Jul 01 16:58:04.629337 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.628028    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/143b1a41-fbb7-467c-8854-ba67eeb0ea3b-var-lib-openvswitch\") pod \"ovnkube-node-j6kft\" (UID: \"143b1a41-fbb7-467c-8854-ba67eeb0ea3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6kft"
Jul 01 16:58:04.629337 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.628047    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/143b1a41-fbb7-467c-8854-ba67eeb0ea3b-host-cni-netd\") pod \"ovnkube-node-j6kft\" (UID: \"143b1a41-fbb7-467c-8854-ba67eeb0ea3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6kft"
Jul 01 16:58:04.629337 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.629051    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7jfww"
Jul 01 16:58:04.629716 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.629699    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 01 16:58:04.629977 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.629956    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-wc49c\""
Jul 01 16:58:04.630051 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.629956    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 01 16:58:04.630051 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.630034    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 01 16:58:04.631290 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.631271    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-x4wf9\""
Jul 01 16:58:04.631587 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.631570    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 01 16:58:04.632008 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.631990    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 01 16:58:04.633075 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.633055    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 01 16:58:04.633336 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.633320    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5bjn2"
Jul 01 16:58:04.635738 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.635718    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bjw8j"
Jul 01 16:58:04.635991 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.635972    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 01 16:58:04.636499 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.636469    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 01 16:58:04.637656 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.636583    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xxs79\""
Jul 01 16:58:04.638538 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.638522    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 01 16:58:04.638625 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.638597    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-nqg6l\""
Jul 01 16:58:04.638733 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.638720    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 01 16:58:04.638849 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.638833    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 01 16:58:04.661057 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.661024    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-5jqdp"
Jul 01 16:58:04.670762 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.670740    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-5jqdp"
Jul 01 16:58:04.695202 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.695153    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-180.ec2.internal" event={"ID":"83e0fb78b992c4b459a7080a59353cae","Type":"ContainerStarted","Data":"6a6bdce092f971643549fdc162c4ef79a1dcf51ef50a3bddea1aead1e3b848c2"}
Jul 01 16:58:04.696310 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.696285    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-180.ec2.internal" event={"ID":"4125034d4754060c768adbf37d115970","Type":"ContainerStarted","Data":"33f89b9cc4561359c65915326f8bf2432dd5af297ea6e529096e070158b5327b"}
Jul 01 16:58:04.715952 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.715930    2563 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 01 16:58:04.728504 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.728473    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0ab87fa2-edd2-4c8b-a666-1efa2c3bae8e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gh58z\" (UID: \"0ab87fa2-edd2-4c8b-a666-1efa2c3bae8e\") " pod="openshift-multus/multus-additional-cni-plugins-gh58z"
Jul 01 16:58:04.728622 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.728518    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/143b1a41-fbb7-467c-8854-ba67eeb0ea3b-etc-openvswitch\") pod \"ovnkube-node-j6kft\" (UID: \"143b1a41-fbb7-467c-8854-ba67eeb0ea3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6kft"
Jul 01 16:58:04.728622 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.728546    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/143b1a41-fbb7-467c-8854-ba67eeb0ea3b-run-openvswitch\") pod \"ovnkube-node-j6kft\" (UID: \"143b1a41-fbb7-467c-8854-ba67eeb0ea3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6kft"
Jul 01 16:58:04.728622 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.728575    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8ef10105-e59e-40eb-a5b7-d6eaa844ce70-etc-sysctl-conf\") pod \"tuned-7jfww\" (UID: \"8ef10105-e59e-40eb-a5b7-d6eaa844ce70\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfww"
Jul 01 16:58:04.728622 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.728618    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/143b1a41-fbb7-467c-8854-ba67eeb0ea3b-env-overrides\") pod \"ovnkube-node-j6kft\" (UID: \"143b1a41-fbb7-467c-8854-ba67eeb0ea3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6kft"
Jul 01 16:58:04.728805 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.728619    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/143b1a41-fbb7-467c-8854-ba67eeb0ea3b-etc-openvswitch\") pod \"ovnkube-node-j6kft\" (UID: \"143b1a41-fbb7-467c-8854-ba67eeb0ea3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6kft"
Jul 01 16:58:04.728805 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.728644    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttgw5\" (UniqueName: \"kubernetes.io/projected/143b1a41-fbb7-467c-8854-ba67eeb0ea3b-kube-api-access-ttgw5\") pod \"ovnkube-node-j6kft\" (UID: \"143b1a41-fbb7-467c-8854-ba67eeb0ea3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6kft"
Jul 01 16:58:04.728805 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.728644    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/143b1a41-fbb7-467c-8854-ba67eeb0ea3b-run-openvswitch\") pod \"ovnkube-node-j6kft\" (UID: \"143b1a41-fbb7-467c-8854-ba67eeb0ea3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6kft"
Jul 01 16:58:04.728805 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.728680    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab1a046b-4d82-418b-9c94-3700fa11dd87-original-pull-secret\") pod \"global-pull-secret-syncer-qp9lx\" (UID: \"ab1a046b-4d82-418b-9c94-3700fa11dd87\") " pod="kube-system/global-pull-secret-syncer-qp9lx"
Jul 01 16:58:04.728805 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.728766    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8ef10105-e59e-40eb-a5b7-d6eaa844ce70-etc-sysctl-d\") pod \"tuned-7jfww\" (UID: \"8ef10105-e59e-40eb-a5b7-d6eaa844ce70\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfww"
Jul 01 16:58:04.728805 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.728798    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8ef10105-e59e-40eb-a5b7-d6eaa844ce70-host\") pod \"tuned-7jfww\" (UID: \"8ef10105-e59e-40eb-a5b7-d6eaa844ce70\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfww"
Jul 01 16:58:04.729088 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.728823    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ef10105-e59e-40eb-a5b7-d6eaa844ce70-tmp\") pod \"tuned-7jfww\" (UID: \"8ef10105-e59e-40eb-a5b7-d6eaa844ce70\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfww"
Jul 01 16:58:04.729088 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.728846    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7ca1dd82-196f-42ef-ac72-d0a1d9f7551f-agent-certs\") pod \"konnectivity-agent-fg79l\" (UID: \"7ca1dd82-196f-42ef-ac72-d0a1d9f7551f\") " pod="kube-system/konnectivity-agent-fg79l"
Jul 01 16:58:04.729088 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.728888    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bf6178b8-7786-4321-9401-ca2ec4315089-serviceca\") pod \"node-ca-bjw8j\" (UID: \"bf6178b8-7786-4321-9401-ca2ec4315089\") " pod="openshift-image-registry/node-ca-bjw8j"
Jul 01 16:58:04.729088 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.728942    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/143b1a41-fbb7-467c-8854-ba67eeb0ea3b-ovn-node-metrics-cert\") pod \"ovnkube-node-j6kft\" (UID: \"143b1a41-fbb7-467c-8854-ba67eeb0ea3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6kft"
Jul 01 16:58:04.729088 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.728987    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/143b1a41-fbb7-467c-8854-ba67eeb0ea3b-ovnkube-script-lib\") pod \"ovnkube-node-j6kft\" (UID: \"143b1a41-fbb7-467c-8854-ba67eeb0ea3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6kft"
Jul 01 16:58:04.729309 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.729110    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0ab87fa2-edd2-4c8b-a666-1efa2c3bae8e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gh58z\" (UID: \"0ab87fa2-edd2-4c8b-a666-1efa2c3bae8e\") " pod="openshift-multus/multus-additional-cni-plugins-gh58z"
Jul 01 16:58:04.729309 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.729135    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0ab87fa2-edd2-4c8b-a666-1efa2c3bae8e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gh58z\" (UID: \"0ab87fa2-edd2-4c8b-a666-1efa2c3bae8e\") " pod="openshift-multus/multus-additional-cni-plugins-gh58z"
Jul 01 16:58:04.729309 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.729164    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/143b1a41-fbb7-467c-8854-ba67eeb0ea3b-host-run-netns\") pod \"ovnkube-node-j6kft\" (UID: \"143b1a41-fbb7-467c-8854-ba67eeb0ea3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6kft"
Jul 01 16:58:04.729309 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.729180    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/143b1a41-fbb7-467c-8854-ba67eeb0ea3b-env-overrides\") pod \"ovnkube-node-j6kft\" (UID: \"143b1a41-fbb7-467c-8854-ba67eeb0ea3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6kft"
Jul 01 16:58:04.729309 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.729204    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/143b1a41-fbb7-467c-8854-ba67eeb0ea3b-host-run-netns\") pod \"ovnkube-node-j6kft\" (UID: \"143b1a41-fbb7-467c-8854-ba67eeb0ea3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6kft"
Jul 01 16:58:04.729309 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.729250    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/143b1a41-fbb7-467c-8854-ba67eeb0ea3b-run-ovn\") pod \"ovnkube-node-j6kft\" (UID: \"143b1a41-fbb7-467c-8854-ba67eeb0ea3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6kft"
Jul 01 16:58:04.729611 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.729320    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/143b1a41-fbb7-467c-8854-ba67eeb0ea3b-node-log\") pod \"ovnkube-node-j6kft\" (UID: \"143b1a41-fbb7-467c-8854-ba67eeb0ea3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6kft"
Jul 01 16:58:04.729611 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.729368    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/143b1a41-fbb7-467c-8854-ba67eeb0ea3b-run-ovn\") pod \"ovnkube-node-j6kft\" (UID: \"143b1a41-fbb7-467c-8854-ba67eeb0ea3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6kft"
Jul 01 16:58:04.729611 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.729395    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/143b1a41-fbb7-467c-8854-ba67eeb0ea3b-host-cni-bin\") pod \"ovnkube-node-j6kft\" (UID: \"143b1a41-fbb7-467c-8854-ba67eeb0ea3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6kft"
Jul 01 16:58:04.729611 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.729419    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/143b1a41-fbb7-467c-8854-ba67eeb0ea3b-node-log\") pod \"ovnkube-node-j6kft\" (UID: \"143b1a41-fbb7-467c-8854-ba67eeb0ea3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6kft"
Jul 01 16:58:04.729611 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.729476    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/143b1a41-fbb7-467c-8854-ba67eeb0ea3b-ovnkube-script-lib\") pod \"ovnkube-node-j6kft\" (UID: \"143b1a41-fbb7-467c-8854-ba67eeb0ea3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6kft"
Jul 01 16:58:04.729611 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.729460    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g78k\" (UniqueName: \"kubernetes.io/projected/bf6178b8-7786-4321-9401-ca2ec4315089-kube-api-access-9g78k\") pod \"node-ca-bjw8j\" (UID: \"bf6178b8-7786-4321-9401-ca2ec4315089\") " pod="openshift-image-registry/node-ca-bjw8j"
Jul 01 16:58:04.729611 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.729538    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/143b1a41-fbb7-467c-8854-ba67eeb0ea3b-host-cni-bin\") pod \"ovnkube-node-j6kft\" (UID: \"143b1a41-fbb7-467c-8854-ba67eeb0ea3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6kft"
Jul 01 16:58:04.729611 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.729591    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ce4b7fc4-39cb-419a-aae4-30587a901329-multus-conf-dir\") pod \"multus-st8xw\" (UID: \"ce4b7fc4-39cb-419a-aae4-30587a901329\") " pod="openshift-multus/multus-st8xw"
Jul 01 16:58:04.729611 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.729613    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkwck\" (UniqueName: \"kubernetes.io/projected/d4b74cf0-0931-4466-bc62-5b807071e600-kube-api-access-bkwck\") pod \"network-check-target-txjpp\" (UID: \"d4b74cf0-0931-4466-bc62-5b807071e600\") " pod="openshift-network-diagnostics/network-check-target-txjpp"
Jul 01 16:58:04.729975 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.729631    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8ef10105-e59e-40eb-a5b7-d6eaa844ce70-etc-systemd\") pod \"tuned-7jfww\" (UID: \"8ef10105-e59e-40eb-a5b7-d6eaa844ce70\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfww"
Jul 01 16:58:04.729975 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.729659    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ce4b7fc4-39cb-419a-aae4-30587a901329-multus-conf-dir\") pod \"multus-st8xw\" (UID: \"ce4b7fc4-39cb-419a-aae4-30587a901329\") " pod="openshift-multus/multus-st8xw"
Jul 01 16:58:04.729975 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.729702    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0ab87fa2-edd2-4c8b-a666-1efa2c3bae8e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gh58z\" (UID: \"0ab87fa2-edd2-4c8b-a666-1efa2c3bae8e\") " pod="openshift-multus/multus-additional-cni-plugins-gh58z"
Jul 01 16:58:04.729975 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.729732    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8ef10105-e59e-40eb-a5b7-d6eaa844ce70-run\") pod \"tuned-7jfww\" (UID: \"8ef10105-e59e-40eb-a5b7-d6eaa844ce70\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfww"
Jul 01 16:58:04.729975 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.729750    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8ef10105-e59e-40eb-a5b7-d6eaa844ce70-lib-modules\") pod \"tuned-7jfww\" (UID: \"8ef10105-e59e-40eb-a5b7-d6eaa844ce70\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfww"
Jul 01 16:58:04.729975 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.729766    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6f94893d-4fd4-4745-9f71-b89dad4d1582-host-slash\") pod \"iptables-alerter-fdj7p\" (UID: \"6f94893d-4fd4-4745-9f71-b89dad4d1582\") " pod="openshift-network-operator/iptables-alerter-fdj7p"
Jul 01 16:58:04.729975 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.729782    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce4b7fc4-39cb-419a-aae4-30587a901329-etc-kubernetes\") pod \"multus-st8xw\" (UID: \"ce4b7fc4-39cb-419a-aae4-30587a901329\") " pod="openshift-multus/multus-st8xw"
Jul 01 16:58:04.729975 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.729803    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rjkh\" (UniqueName: \"kubernetes.io/projected/ce4b7fc4-39cb-419a-aae4-30587a901329-kube-api-access-5rjkh\") pod \"multus-st8xw\" (UID: \"ce4b7fc4-39cb-419a-aae4-30587a901329\") " pod="openshift-multus/multus-st8xw"
Jul 01 16:58:04.729975 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.729818    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ab87fa2-edd2-4c8b-a666-1efa2c3bae8e-system-cni-dir\") pod \"multus-additional-cni-plugins-gh58z\" (UID: \"0ab87fa2-edd2-4c8b-a666-1efa2c3bae8e\") " pod="openshift-multus/multus-additional-cni-plugins-gh58z"
Jul 01 16:58:04.729975 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.729833    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/143b1a41-fbb7-467c-8854-ba67eeb0ea3b-host-slash\") pod \"ovnkube-node-j6kft\" (UID: \"143b1a41-fbb7-467c-8854-ba67eeb0ea3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6kft"
Jul 01 16:58:04.729975 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.729843    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce4b7fc4-39cb-419a-aae4-30587a901329-etc-kubernetes\") pod \"multus-st8xw\" (UID: \"ce4b7fc4-39cb-419a-aae4-30587a901329\") " pod="openshift-multus/multus-st8xw"
Jul 01 16:58:04.729975 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.729852    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6f94893d-4fd4-4745-9f71-b89dad4d1582-host-slash\") pod \"iptables-alerter-fdj7p\" (UID: \"6f94893d-4fd4-4745-9f71-b89dad4d1582\") " pod="openshift-network-operator/iptables-alerter-fdj7p"
Jul 01 16:58:04.729975 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.729847    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/143b1a41-fbb7-467c-8854-ba67eeb0ea3b-run-systemd\") pod \"ovnkube-node-j6kft\" (UID: \"143b1a41-fbb7-467c-8854-ba67eeb0ea3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6kft"
Jul 01 16:58:04.729975 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.729882    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/143b1a41-fbb7-467c-8854-ba67eeb0ea3b-host-slash\") pod \"ovnkube-node-j6kft\" (UID: \"143b1a41-fbb7-467c-8854-ba67eeb0ea3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6kft"
Jul 01 16:58:04.729975 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.729877    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ab87fa2-edd2-4c8b-a666-1efa2c3bae8e-system-cni-dir\") pod \"multus-additional-cni-plugins-gh58z\" (UID: \"0ab87fa2-edd2-4c8b-a666-1efa2c3bae8e\") " pod="openshift-multus/multus-additional-cni-plugins-gh58z"
Jul 01 16:58:04.729975 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.729908    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/143b1a41-fbb7-467c-8854-ba67eeb0ea3b-host-cni-netd\") pod \"ovnkube-node-j6kft\" (UID: \"143b1a41-fbb7-467c-8854-ba67eeb0ea3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6kft"
Jul 01 16:58:04.729975 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.729909    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/143b1a41-fbb7-467c-8854-ba67eeb0ea3b-run-systemd\") pod \"ovnkube-node-j6kft\" (UID: \"143b1a41-fbb7-467c-8854-ba67eeb0ea3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6kft"
Jul 01 16:58:04.730634 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.729936    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8ef10105-e59e-40eb-a5b7-d6eaa844ce70-etc-sysconfig\") pod \"tuned-7jfww\" (UID: \"8ef10105-e59e-40eb-a5b7-d6eaa844ce70\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfww"
Jul 01 16:58:04.730634 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.729953    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/143b1a41-fbb7-467c-8854-ba67eeb0ea3b-host-cni-netd\") pod \"ovnkube-node-j6kft\" (UID: \"143b1a41-fbb7-467c-8854-ba67eeb0ea3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6kft"
Jul 01 16:58:04.730634 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.729962    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ce4b7fc4-39cb-419a-aae4-30587a901329-cnibin\") pod \"multus-st8xw\" (UID: \"ce4b7fc4-39cb-419a-aae4-30587a901329\") " pod="openshift-multus/multus-st8xw"
Jul 01 16:58:04.730634 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.729992    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ce4b7fc4-39cb-419a-aae4-30587a901329-hostroot\") pod \"multus-st8xw\" (UID: \"ce4b7fc4-39cb-419a-aae4-30587a901329\") " pod="openshift-multus/multus-st8xw"
Jul 01 16:58:04.730634 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.730011    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ce4b7fc4-39cb-419a-aae4-30587a901329-cnibin\") pod \"multus-st8xw\" (UID: \"ce4b7fc4-39cb-419a-aae4-30587a901329\") " pod="openshift-multus/multus-st8xw"
Jul 01 16:58:04.730634 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.730008    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/899ca2bf-8e98-4def-a2f9-ace75d3170a0-metrics-certs\") pod \"network-metrics-daemon-fp8j8\" (UID: \"899ca2bf-8e98-4def-a2f9-ace75d3170a0\") " pod="openshift-multus/network-metrics-daemon-fp8j8"
Jul 01 16:58:04.730634 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.730061    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ce4b7fc4-39cb-419a-aae4-30587a901329-hostroot\") pod \"multus-st8xw\" (UID: \"ce4b7fc4-39cb-419a-aae4-30587a901329\") " pod="openshift-multus/multus-st8xw"
Jul 01 16:58:04.730634 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:04.730101    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 16:58:04.730634 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.730115    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/143b1a41-fbb7-467c-8854-ba67eeb0ea3b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j6kft\" (UID: \"143b1a41-fbb7-467c-8854-ba67eeb0ea3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6kft"
Jul 01 16:58:04.730634 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.730143    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ef10105-e59e-40eb-a5b7-d6eaa844ce70-etc-kubernetes\") pod \"tuned-7jfww\" (UID: \"8ef10105-e59e-40eb-a5b7-d6eaa844ce70\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfww"
Jul 01 16:58:04.730634 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:04.730168    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/899ca2bf-8e98-4def-a2f9-ace75d3170a0-metrics-certs podName:899ca2bf-8e98-4def-a2f9-ace75d3170a0 nodeName:}" failed. No retries permitted until 2026-07-01 16:58:05.230138818 +0000 UTC m=+3.027277198 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/899ca2bf-8e98-4def-a2f9-ace75d3170a0-metrics-certs") pod "network-metrics-daemon-fp8j8" (UID: "899ca2bf-8e98-4def-a2f9-ace75d3170a0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 16:58:04.730634 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.730180    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/143b1a41-fbb7-467c-8854-ba67eeb0ea3b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j6kft\" (UID: \"143b1a41-fbb7-467c-8854-ba67eeb0ea3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6kft"
Jul 01 16:58:04.730634 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.730209    2563 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 01 16:58:04.730634 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.730230    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/786cf9b6-b11a-4572-9517-b013acc6ae5c-device-dir\") pod \"aws-ebs-csi-driver-node-h79ks\" (UID: \"786cf9b6-b11a-4572-9517-b013acc6ae5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h79ks"
Jul 01 16:58:04.730634 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.730297    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbncl\" (UniqueName: \"kubernetes.io/projected/8ef10105-e59e-40eb-a5b7-d6eaa844ce70-kube-api-access-tbncl\") pod \"tuned-7jfww\" (UID: \"8ef10105-e59e-40eb-a5b7-d6eaa844ce70\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfww"
Jul 01 16:58:04.730634 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.730319    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x42d9\" (UniqueName: \"kubernetes.io/projected/242fcfc2-df03-4f81-bd5c-1bf8ccf2d2d4-kube-api-access-x42d9\") pod \"node-resolver-5bjn2\" (UID: \"242fcfc2-df03-4f81-bd5c-1bf8ccf2d2d4\") " pod="openshift-dns/node-resolver-5bjn2"
Jul 01 16:58:04.730634 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.730336    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ce4b7fc4-39cb-419a-aae4-30587a901329-system-cni-dir\") pod \"multus-st8xw\" (UID: \"ce4b7fc4-39cb-419a-aae4-30587a901329\") " pod="openshift-multus/multus-st8xw"
Jul 01 16:58:04.731600 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.730379    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/143b1a41-fbb7-467c-8854-ba67eeb0ea3b-host-run-ovn-kubernetes\") pod \"ovnkube-node-j6kft\" (UID: \"143b1a41-fbb7-467c-8854-ba67eeb0ea3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6kft"
Jul 01 16:58:04.731600 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.730394    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ce4b7fc4-39cb-419a-aae4-30587a901329-system-cni-dir\") pod \"multus-st8xw\" (UID: \"ce4b7fc4-39cb-419a-aae4-30587a901329\") " pod="openshift-multus/multus-st8xw"
Jul 01 16:58:04.731600 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.730395    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8ef10105-e59e-40eb-a5b7-d6eaa844ce70-sys\") pod \"tuned-7jfww\" (UID: \"8ef10105-e59e-40eb-a5b7-d6eaa844ce70\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfww"
Jul 01 16:58:04.731600 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.730420    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/143b1a41-fbb7-467c-8854-ba67eeb0ea3b-host-run-ovn-kubernetes\") pod \"ovnkube-node-j6kft\" (UID: \"143b1a41-fbb7-467c-8854-ba67eeb0ea3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6kft"
Jul 01 16:58:04.731600 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.730428    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ce4b7fc4-39cb-419a-aae4-30587a901329-cni-binary-copy\") pod \"multus-st8xw\" (UID: \"ce4b7fc4-39cb-419a-aae4-30587a901329\") " pod="openshift-multus/multus-st8xw"
Jul 01 16:58:04.731600 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.730444    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce4b7fc4-39cb-419a-aae4-30587a901329-host-var-lib-kubelet\") pod \"multus-st8xw\" (UID: \"ce4b7fc4-39cb-419a-aae4-30587a901329\") " pod="openshift-multus/multus-st8xw"
Jul 01 16:58:04.731600 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.730466    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ef10105-e59e-40eb-a5b7-d6eaa844ce70-var-lib-kubelet\") pod \"tuned-7jfww\" (UID: \"8ef10105-e59e-40eb-a5b7-d6eaa844ce70\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfww"
Jul 01 16:58:04.731600 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.730491    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/786cf9b6-b11a-4572-9517-b013acc6ae5c-socket-dir\") pod \"aws-ebs-csi-driver-node-h79ks\" (UID: \"786cf9b6-b11a-4572-9517-b013acc6ae5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h79ks"
Jul 01 16:58:04.731600 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.730514    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/786cf9b6-b11a-4572-9517-b013acc6ae5c-sys-fs\") pod \"aws-ebs-csi-driver-node-h79ks\" (UID: \"786cf9b6-b11a-4572-9517-b013acc6ae5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h79ks"
Jul 01 16:58:04.731600 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.730506    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce4b7fc4-39cb-419a-aae4-30587a901329-host-var-lib-kubelet\") pod \"multus-st8xw\" (UID: \"ce4b7fc4-39cb-419a-aae4-30587a901329\") " pod="openshift-multus/multus-st8xw"
Jul 01 16:58:04.731600 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.730544    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ce4b7fc4-39cb-419a-aae4-30587a901329-host-var-lib-cni-multus\") pod \"multus-st8xw\" (UID: \"ce4b7fc4-39cb-419a-aae4-30587a901329\") " pod="openshift-multus/multus-st8xw"
Jul 01 16:58:04.731600 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.730574    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0ab87fa2-edd2-4c8b-a666-1efa2c3bae8e-cnibin\") pod \"multus-additional-cni-plugins-gh58z\" (UID: \"0ab87fa2-edd2-4c8b-a666-1efa2c3bae8e\") " pod="openshift-multus/multus-additional-cni-plugins-gh58z"
Jul 01 16:58:04.731600 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.730620    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8ef10105-e59e-40eb-a5b7-d6eaa844ce70-etc-modprobe-d\") pod \"tuned-7jfww\" (UID: \"8ef10105-e59e-40eb-a5b7-d6eaa844ce70\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfww"
Jul 01 16:58:04.731600 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.730635    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ce4b7fc4-39cb-419a-aae4-30587a901329-host-var-lib-cni-multus\") pod \"multus-st8xw\" (UID: \"ce4b7fc4-39cb-419a-aae4-30587a901329\") " pod="openshift-multus/multus-st8xw"
Jul 01 16:58:04.731600 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.730647    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z52b2\" (UniqueName: \"kubernetes.io/projected/786cf9b6-b11a-4572-9517-b013acc6ae5c-kube-api-access-z52b2\") pod \"aws-ebs-csi-driver-node-h79ks\" (UID: \"786cf9b6-b11a-4572-9517-b013acc6ae5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h79ks"
Jul 01 16:58:04.731600 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.730672    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0ab87fa2-edd2-4c8b-a666-1efa2c3bae8e-cnibin\") pod \"multus-additional-cni-plugins-gh58z\" (UID: \"0ab87fa2-edd2-4c8b-a666-1efa2c3bae8e\") " pod="openshift-multus/multus-additional-cni-plugins-gh58z"
Jul 01 16:58:04.731600 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.730686    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6f94893d-4fd4-4745-9f71-b89dad4d1582-iptables-alerter-script\") pod \"iptables-alerter-fdj7p\" (UID: \"6f94893d-4fd4-4745-9f71-b89dad4d1582\") " pod="openshift-network-operator/iptables-alerter-fdj7p"
Jul 01 16:58:04.732473 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.730708    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ce4b7fc4-39cb-419a-aae4-30587a901329-multus-socket-dir-parent\") pod \"multus-st8xw\" (UID: \"ce4b7fc4-39cb-419a-aae4-30587a901329\") " pod="openshift-multus/multus-st8xw"
Jul 01 16:58:04.732473 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.730725    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkjkt\" (UniqueName: \"kubernetes.io/projected/0ab87fa2-edd2-4c8b-a666-1efa2c3bae8e-kube-api-access-vkjkt\") pod \"multus-additional-cni-plugins-gh58z\" (UID: \"0ab87fa2-edd2-4c8b-a666-1efa2c3bae8e\") " pod="openshift-multus/multus-additional-cni-plugins-gh58z"
Jul 01 16:58:04.732473 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.730740    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnkfn\" (UniqueName: \"kubernetes.io/projected/6f94893d-4fd4-4745-9f71-b89dad4d1582-kube-api-access-pnkfn\") pod \"iptables-alerter-fdj7p\" (UID: \"6f94893d-4fd4-4745-9f71-b89dad4d1582\") " pod="openshift-network-operator/iptables-alerter-fdj7p"
Jul 01 16:58:04.732473 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.730764    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ce4b7fc4-39cb-419a-aae4-30587a901329-multus-cni-dir\") pod \"multus-st8xw\" (UID: \"ce4b7fc4-39cb-419a-aae4-30587a901329\") " pod="openshift-multus/multus-st8xw"
Jul 01 16:58:04.732473 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.730786    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ce4b7fc4-39cb-419a-aae4-30587a901329-host-var-lib-cni-bin\") pod \"multus-st8xw\" (UID: \"ce4b7fc4-39cb-419a-aae4-30587a901329\") " pod="openshift-multus/multus-st8xw"
Jul 01 16:58:04.732473 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.730809    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ce4b7fc4-39cb-419a-aae4-30587a901329-host-run-multus-certs\") pod \"multus-st8xw\" (UID: \"ce4b7fc4-39cb-419a-aae4-30587a901329\") " pod="openshift-multus/multus-st8xw"
Jul 01 16:58:04.732473 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.730837    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0ab87fa2-edd2-4c8b-a666-1efa2c3bae8e-os-release\") pod \"multus-additional-cni-plugins-gh58z\" (UID: \"0ab87fa2-edd2-4c8b-a666-1efa2c3bae8e\") " pod="openshift-multus/multus-additional-cni-plugins-gh58z"
Jul 01 16:58:04.732473 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.730859    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7ca1dd82-196f-42ef-ac72-d0a1d9f7551f-konnectivity-ca\") pod \"konnectivity-agent-fg79l\" (UID: \"7ca1dd82-196f-42ef-ac72-d0a1d9f7551f\") " pod="kube-system/konnectivity-agent-fg79l"
Jul 01 16:58:04.732473 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.730881    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/786cf9b6-b11a-4572-9517-b013acc6ae5c-etc-selinux\") pod \"aws-ebs-csi-driver-node-h79ks\" (UID: \"786cf9b6-b11a-4572-9517-b013acc6ae5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h79ks"
Jul 01 16:58:04.732473 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.730891    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ce4b7fc4-39cb-419a-aae4-30587a901329-cni-binary-copy\") pod \"multus-st8xw\" (UID: \"ce4b7fc4-39cb-419a-aae4-30587a901329\") " pod="openshift-multus/multus-st8xw"
Jul 01 16:58:04.732473 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.730905    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/242fcfc2-df03-4f81-bd5c-1bf8ccf2d2d4-hosts-file\") pod \"node-resolver-5bjn2\" (UID: \"242fcfc2-df03-4f81-bd5c-1bf8ccf2d2d4\") " pod="openshift-dns/node-resolver-5bjn2"
Jul 01 16:58:04.732473 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.730951    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0ab87fa2-edd2-4c8b-a666-1efa2c3bae8e-cni-binary-copy\") pod \"multus-additional-cni-plugins-gh58z\" (UID: \"0ab87fa2-edd2-4c8b-a666-1efa2c3bae8e\") " pod="openshift-multus/multus-additional-cni-plugins-gh58z"
Jul 01 16:58:04.732473 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.730957    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ce4b7fc4-39cb-419a-aae4-30587a901329-host-run-multus-certs\") pod \"multus-st8xw\" (UID: \"ce4b7fc4-39cb-419a-aae4-30587a901329\") " pod="openshift-multus/multus-st8xw"
Jul 01 16:58:04.732473 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.730976    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/143b1a41-fbb7-467c-8854-ba67eeb0ea3b-host-kubelet\") pod \"ovnkube-node-j6kft\" (UID: \"143b1a41-fbb7-467c-8854-ba67eeb0ea3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6kft"
Jul 01 16:58:04.732473 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.731032    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/143b1a41-fbb7-467c-8854-ba67eeb0ea3b-host-kubelet\") pod \"ovnkube-node-j6kft\" (UID: \"143b1a41-fbb7-467c-8854-ba67eeb0ea3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6kft"
Jul 01 16:58:04.732473 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.731042    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0ab87fa2-edd2-4c8b-a666-1efa2c3bae8e-os-release\") pod \"multus-additional-cni-plugins-gh58z\" (UID: \"0ab87fa2-edd2-4c8b-a666-1efa2c3bae8e\") " pod="openshift-multus/multus-additional-cni-plugins-gh58z"
Jul 01 16:58:04.732473 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.731118    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ce4b7fc4-39cb-419a-aae4-30587a901329-multus-cni-dir\") pod \"multus-st8xw\" (UID: \"ce4b7fc4-39cb-419a-aae4-30587a901329\") " pod="openshift-multus/multus-st8xw"
Jul 01 16:58:04.733338 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.731154    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/143b1a41-fbb7-467c-8854-ba67eeb0ea3b-var-lib-openvswitch\") pod \"ovnkube-node-j6kft\" (UID: \"143b1a41-fbb7-467c-8854-ba67eeb0ea3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6kft"
Jul 01 16:58:04.733338 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.731164    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ce4b7fc4-39cb-419a-aae4-30587a901329-host-var-lib-cni-bin\") pod \"multus-st8xw\" (UID: \"ce4b7fc4-39cb-419a-aae4-30587a901329\") " pod="openshift-multus/multus-st8xw"
Jul 01 16:58:04.733338 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.731193    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/143b1a41-fbb7-467c-8854-ba67eeb0ea3b-var-lib-openvswitch\") pod \"ovnkube-node-j6kft\" (UID: \"143b1a41-fbb7-467c-8854-ba67eeb0ea3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6kft"
Jul 01 16:58:04.733338 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.731218    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ab1a046b-4d82-418b-9c94-3700fa11dd87-kubelet-config\") pod \"global-pull-secret-syncer-qp9lx\" (UID: \"ab1a046b-4d82-418b-9c94-3700fa11dd87\") " pod="kube-system/global-pull-secret-syncer-qp9lx"
Jul 01 16:58:04.733338 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.731244    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8ef10105-e59e-40eb-a5b7-d6eaa844ce70-etc-tuned\") pod \"tuned-7jfww\" (UID: \"8ef10105-e59e-40eb-a5b7-d6eaa844ce70\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfww"
Jul 01 16:58:04.733338 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.731247    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ce4b7fc4-39cb-419a-aae4-30587a901329-multus-socket-dir-parent\") pod \"multus-st8xw\" (UID: \"ce4b7fc4-39cb-419a-aae4-30587a901329\") " pod="openshift-multus/multus-st8xw"
Jul 01 16:58:04.733338 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.731271    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gv8z\" (UniqueName: \"kubernetes.io/projected/899ca2bf-8e98-4def-a2f9-ace75d3170a0-kube-api-access-7gv8z\") pod \"network-metrics-daemon-fp8j8\" (UID: \"899ca2bf-8e98-4def-a2f9-ace75d3170a0\") " pod="openshift-multus/network-metrics-daemon-fp8j8"
Jul 01 16:58:04.733338 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.731275    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6f94893d-4fd4-4745-9f71-b89dad4d1582-iptables-alerter-script\") pod \"iptables-alerter-fdj7p\" (UID: \"6f94893d-4fd4-4745-9f71-b89dad4d1582\") " pod="openshift-network-operator/iptables-alerter-fdj7p"
Jul 01 16:58:04.733338 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.731289    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bf6178b8-7786-4321-9401-ca2ec4315089-host\") pod \"node-ca-bjw8j\" (UID: \"bf6178b8-7786-4321-9401-ca2ec4315089\") " pod="openshift-image-registry/node-ca-bjw8j"
Jul 01 16:58:04.733338 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.731327    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/143b1a41-fbb7-467c-8854-ba67eeb0ea3b-log-socket\") pod \"ovnkube-node-j6kft\" (UID: \"143b1a41-fbb7-467c-8854-ba67eeb0ea3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6kft"
Jul 01 16:58:04.733338 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.731375    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/143b1a41-fbb7-467c-8854-ba67eeb0ea3b-ovnkube-config\") pod \"ovnkube-node-j6kft\" (UID: \"143b1a41-fbb7-467c-8854-ba67eeb0ea3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6kft"
Jul 01 16:58:04.733338 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.731395    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/143b1a41-fbb7-467c-8854-ba67eeb0ea3b-log-socket\") pod \"ovnkube-node-j6kft\" (UID: \"143b1a41-fbb7-467c-8854-ba67eeb0ea3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6kft"
Jul 01 16:58:04.733338 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.731401    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/786cf9b6-b11a-4572-9517-b013acc6ae5c-registration-dir\") pod \"aws-ebs-csi-driver-node-h79ks\" (UID: \"786cf9b6-b11a-4572-9517-b013acc6ae5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h79ks"
Jul 01 16:58:04.733338 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.731442    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/242fcfc2-df03-4f81-bd5c-1bf8ccf2d2d4-tmp-dir\") pod \"node-resolver-5bjn2\" (UID: \"242fcfc2-df03-4f81-bd5c-1bf8ccf2d2d4\") " pod="openshift-dns/node-resolver-5bjn2"
Jul 01 16:58:04.733338 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.731480    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ce4b7fc4-39cb-419a-aae4-30587a901329-multus-daemon-config\") pod \"multus-st8xw\" (UID: \"ce4b7fc4-39cb-419a-aae4-30587a901329\") " pod="openshift-multus/multus-st8xw"
Jul 01 16:58:04.733338 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.731494    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0ab87fa2-edd2-4c8b-a666-1efa2c3bae8e-cni-binary-copy\") pod \"multus-additional-cni-plugins-gh58z\" (UID: \"0ab87fa2-edd2-4c8b-a666-1efa2c3bae8e\") " pod="openshift-multus/multus-additional-cni-plugins-gh58z"
Jul 01 16:58:04.733338 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.731504    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/143b1a41-fbb7-467c-8854-ba67eeb0ea3b-systemd-units\") pod \"ovnkube-node-j6kft\" (UID: \"143b1a41-fbb7-467c-8854-ba67eeb0ea3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6kft"
Jul 01 16:58:04.734562 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.731528    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ab1a046b-4d82-418b-9c94-3700fa11dd87-dbus\") pod \"global-pull-secret-syncer-qp9lx\" (UID: \"ab1a046b-4d82-418b-9c94-3700fa11dd87\") " pod="kube-system/global-pull-secret-syncer-qp9lx"
Jul 01 16:58:04.734562 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.731559    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/786cf9b6-b11a-4572-9517-b013acc6ae5c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-h79ks\" (UID: \"786cf9b6-b11a-4572-9517-b013acc6ae5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h79ks"
Jul 01 16:58:04.734562 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.731585    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ce4b7fc4-39cb-419a-aae4-30587a901329-os-release\") pod \"multus-st8xw\" (UID: \"ce4b7fc4-39cb-419a-aae4-30587a901329\") " pod="openshift-multus/multus-st8xw"
Jul 01 16:58:04.734562 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.731560    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/143b1a41-fbb7-467c-8854-ba67eeb0ea3b-systemd-units\") pod \"ovnkube-node-j6kft\" (UID: \"143b1a41-fbb7-467c-8854-ba67eeb0ea3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6kft"
Jul 01 16:58:04.734562 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.731611    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ce4b7fc4-39cb-419a-aae4-30587a901329-host-run-k8s-cni-cncf-io\") pod \"multus-st8xw\" (UID: \"ce4b7fc4-39cb-419a-aae4-30587a901329\") " pod="openshift-multus/multus-st8xw"
Jul 01 16:58:04.734562 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.731638    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ce4b7fc4-39cb-419a-aae4-30587a901329-host-run-netns\") pod \"multus-st8xw\" (UID: \"ce4b7fc4-39cb-419a-aae4-30587a901329\") " pod="openshift-multus/multus-st8xw"
Jul 01 16:58:04.734562 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.731661    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0ab87fa2-edd2-4c8b-a666-1efa2c3bae8e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gh58z\" (UID: \"0ab87fa2-edd2-4c8b-a666-1efa2c3bae8e\") " pod="openshift-multus/multus-additional-cni-plugins-gh58z"
Jul 01 16:58:04.734562 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.731665    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ce4b7fc4-39cb-419a-aae4-30587a901329-os-release\") pod \"multus-st8xw\" (UID: \"ce4b7fc4-39cb-419a-aae4-30587a901329\") " pod="openshift-multus/multus-st8xw"
Jul 01 16:58:04.734562 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.731761    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ce4b7fc4-39cb-419a-aae4-30587a901329-host-run-netns\") pod \"multus-st8xw\" (UID: \"ce4b7fc4-39cb-419a-aae4-30587a901329\") " pod="openshift-multus/multus-st8xw"
Jul 01 16:58:04.734562 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.731865    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ce4b7fc4-39cb-419a-aae4-30587a901329-host-run-k8s-cni-cncf-io\") pod \"multus-st8xw\" (UID: \"ce4b7fc4-39cb-419a-aae4-30587a901329\") " pod="openshift-multus/multus-st8xw"
Jul 01 16:58:04.734562 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.731894    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0ab87fa2-edd2-4c8b-a666-1efa2c3bae8e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gh58z\" (UID: \"0ab87fa2-edd2-4c8b-a666-1efa2c3bae8e\") " pod="openshift-multus/multus-additional-cni-plugins-gh58z"
Jul 01 16:58:04.734562 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.731910    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/143b1a41-fbb7-467c-8854-ba67eeb0ea3b-ovnkube-config\") pod \"ovnkube-node-j6kft\" (UID: \"143b1a41-fbb7-467c-8854-ba67eeb0ea3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6kft"
Jul 01 16:58:04.734562 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.731915    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ce4b7fc4-39cb-419a-aae4-30587a901329-multus-daemon-config\") pod \"multus-st8xw\" (UID: \"ce4b7fc4-39cb-419a-aae4-30587a901329\") " pod="openshift-multus/multus-st8xw"
Jul 01 16:58:04.734562 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.734102    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/143b1a41-fbb7-467c-8854-ba67eeb0ea3b-ovn-node-metrics-cert\") pod \"ovnkube-node-j6kft\" (UID: \"143b1a41-fbb7-467c-8854-ba67eeb0ea3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6kft"
Jul 01 16:58:04.738225 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:04.737641    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 16:58:04.738225 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:04.737666    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 16:58:04.738225 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:04.737680    2563 projected.go:196] Error preparing data for projected volume kube-api-access-bkwck for pod openshift-network-diagnostics/network-check-target-txjpp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 16:58:04.738225 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:04.737732    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d4b74cf0-0931-4466-bc62-5b807071e600-kube-api-access-bkwck podName:d4b74cf0-0931-4466-bc62-5b807071e600 nodeName:}" failed. No retries permitted until 2026-07-01 16:58:05.237716293 +0000 UTC m=+3.034854675 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-bkwck" (UniqueName: "kubernetes.io/projected/d4b74cf0-0931-4466-bc62-5b807071e600-kube-api-access-bkwck") pod "network-check-target-txjpp" (UID: "d4b74cf0-0931-4466-bc62-5b807071e600") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 16:58:04.738731 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.738315    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttgw5\" (UniqueName: \"kubernetes.io/projected/143b1a41-fbb7-467c-8854-ba67eeb0ea3b-kube-api-access-ttgw5\") pod \"ovnkube-node-j6kft\" (UID: \"143b1a41-fbb7-467c-8854-ba67eeb0ea3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-j6kft"
Jul 01 16:58:04.739164 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.739141    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rjkh\" (UniqueName: \"kubernetes.io/projected/ce4b7fc4-39cb-419a-aae4-30587a901329-kube-api-access-5rjkh\") pod \"multus-st8xw\" (UID: \"ce4b7fc4-39cb-419a-aae4-30587a901329\") " pod="openshift-multus/multus-st8xw"
Jul 01 16:58:04.741003 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.740590    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnkfn\" (UniqueName: \"kubernetes.io/projected/6f94893d-4fd4-4745-9f71-b89dad4d1582-kube-api-access-pnkfn\") pod \"iptables-alerter-fdj7p\" (UID: \"6f94893d-4fd4-4745-9f71-b89dad4d1582\") " pod="openshift-network-operator/iptables-alerter-fdj7p"
Jul 01 16:58:04.741003 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.740649    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkjkt\" (UniqueName: \"kubernetes.io/projected/0ab87fa2-edd2-4c8b-a666-1efa2c3bae8e-kube-api-access-vkjkt\") pod \"multus-additional-cni-plugins-gh58z\" (UID: \"0ab87fa2-edd2-4c8b-a666-1efa2c3bae8e\") " pod="openshift-multus/multus-additional-cni-plugins-gh58z"
Jul 01 16:58:04.742539 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.742504    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gv8z\" (UniqueName: \"kubernetes.io/projected/899ca2bf-8e98-4def-a2f9-ace75d3170a0-kube-api-access-7gv8z\") pod \"network-metrics-daemon-fp8j8\" (UID: \"899ca2bf-8e98-4def-a2f9-ace75d3170a0\") " pod="openshift-multus/network-metrics-daemon-fp8j8"
Jul 01 16:58:04.832605 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.832570    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8ef10105-e59e-40eb-a5b7-d6eaa844ce70-etc-sysctl-conf\") pod \"tuned-7jfww\" (UID: \"8ef10105-e59e-40eb-a5b7-d6eaa844ce70\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfww"
Jul 01 16:58:04.832605 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.832610    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab1a046b-4d82-418b-9c94-3700fa11dd87-original-pull-secret\") pod \"global-pull-secret-syncer-qp9lx\" (UID: \"ab1a046b-4d82-418b-9c94-3700fa11dd87\") " pod="kube-system/global-pull-secret-syncer-qp9lx"
Jul 01 16:58:04.832840 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.832632    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8ef10105-e59e-40eb-a5b7-d6eaa844ce70-etc-sysctl-d\") pod \"tuned-7jfww\" (UID: \"8ef10105-e59e-40eb-a5b7-d6eaa844ce70\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfww"
Jul 01 16:58:04.832840 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.832653    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8ef10105-e59e-40eb-a5b7-d6eaa844ce70-host\") pod \"tuned-7jfww\" (UID: \"8ef10105-e59e-40eb-a5b7-d6eaa844ce70\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfww"
Jul 01 16:58:04.832840 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.832679    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ef10105-e59e-40eb-a5b7-d6eaa844ce70-tmp\") pod \"tuned-7jfww\" (UID: \"8ef10105-e59e-40eb-a5b7-d6eaa844ce70\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfww"
Jul 01 16:58:04.832840 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.832701    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7ca1dd82-196f-42ef-ac72-d0a1d9f7551f-agent-certs\") pod \"konnectivity-agent-fg79l\" (UID: \"7ca1dd82-196f-42ef-ac72-d0a1d9f7551f\") " pod="kube-system/konnectivity-agent-fg79l"
Jul 01 16:58:04.832840 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.832721    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bf6178b8-7786-4321-9401-ca2ec4315089-serviceca\") pod \"node-ca-bjw8j\" (UID: \"bf6178b8-7786-4321-9401-ca2ec4315089\") " pod="openshift-image-registry/node-ca-bjw8j"
Jul 01 16:58:04.832840 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.832748    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g78k\" (UniqueName: \"kubernetes.io/projected/bf6178b8-7786-4321-9401-ca2ec4315089-kube-api-access-9g78k\") pod \"node-ca-bjw8j\" (UID: \"bf6178b8-7786-4321-9401-ca2ec4315089\") " pod="openshift-image-registry/node-ca-bjw8j"
Jul 01 16:58:04.832840 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.832780    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8ef10105-e59e-40eb-a5b7-d6eaa844ce70-etc-systemd\") pod \"tuned-7jfww\" (UID: \"8ef10105-e59e-40eb-a5b7-d6eaa844ce70\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfww"
Jul 01 16:58:04.832840 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.832799    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8ef10105-e59e-40eb-a5b7-d6eaa844ce70-run\") pod \"tuned-7jfww\" (UID: \"8ef10105-e59e-40eb-a5b7-d6eaa844ce70\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfww"
Jul 01 16:58:04.832840 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.832825    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8ef10105-e59e-40eb-a5b7-d6eaa844ce70-lib-modules\") pod \"tuned-7jfww\" (UID: \"8ef10105-e59e-40eb-a5b7-d6eaa844ce70\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfww"
Jul 01 16:58:04.833263 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.832852    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8ef10105-e59e-40eb-a5b7-d6eaa844ce70-etc-sysconfig\") pod \"tuned-7jfww\" (UID: \"8ef10105-e59e-40eb-a5b7-d6eaa844ce70\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfww"
Jul 01 16:58:04.833263 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.832888    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ef10105-e59e-40eb-a5b7-d6eaa844ce70-etc-kubernetes\") pod \"tuned-7jfww\" (UID: \"8ef10105-e59e-40eb-a5b7-d6eaa844ce70\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfww"
Jul 01 16:58:04.833263 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.832911    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/786cf9b6-b11a-4572-9517-b013acc6ae5c-device-dir\") pod \"aws-ebs-csi-driver-node-h79ks\" (UID: \"786cf9b6-b11a-4572-9517-b013acc6ae5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h79ks"
Jul 01 16:58:04.833263 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.832935    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbncl\" (UniqueName: \"kubernetes.io/projected/8ef10105-e59e-40eb-a5b7-d6eaa844ce70-kube-api-access-tbncl\") pod \"tuned-7jfww\" (UID: \"8ef10105-e59e-40eb-a5b7-d6eaa844ce70\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfww"
Jul 01 16:58:04.833263 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.832959    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x42d9\" (UniqueName: \"kubernetes.io/projected/242fcfc2-df03-4f81-bd5c-1bf8ccf2d2d4-kube-api-access-x42d9\") pod \"node-resolver-5bjn2\" (UID: \"242fcfc2-df03-4f81-bd5c-1bf8ccf2d2d4\") " pod="openshift-dns/node-resolver-5bjn2"
Jul 01 16:58:04.833263 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.832984    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8ef10105-e59e-40eb-a5b7-d6eaa844ce70-sys\") pod \"tuned-7jfww\" (UID: \"8ef10105-e59e-40eb-a5b7-d6eaa844ce70\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfww"
Jul 01 16:58:04.833263 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.833009    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ef10105-e59e-40eb-a5b7-d6eaa844ce70-var-lib-kubelet\") pod \"tuned-7jfww\" (UID: \"8ef10105-e59e-40eb-a5b7-d6eaa844ce70\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfww"
Jul 01 16:58:04.833263 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.833032    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/786cf9b6-b11a-4572-9517-b013acc6ae5c-socket-dir\") pod \"aws-ebs-csi-driver-node-h79ks\" (UID: \"786cf9b6-b11a-4572-9517-b013acc6ae5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h79ks"
Jul 01 16:58:04.833263 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.833054    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/786cf9b6-b11a-4572-9517-b013acc6ae5c-sys-fs\") pod \"aws-ebs-csi-driver-node-h79ks\" (UID: \"786cf9b6-b11a-4572-9517-b013acc6ae5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h79ks"
Jul 01 16:58:04.833263 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.833081    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8ef10105-e59e-40eb-a5b7-d6eaa844ce70-etc-modprobe-d\") pod \"tuned-7jfww\" (UID: \"8ef10105-e59e-40eb-a5b7-d6eaa844ce70\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfww"
Jul 01 16:58:04.833263 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.833112    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z52b2\" (UniqueName: \"kubernetes.io/projected/786cf9b6-b11a-4572-9517-b013acc6ae5c-kube-api-access-z52b2\") pod \"aws-ebs-csi-driver-node-h79ks\" (UID: \"786cf9b6-b11a-4572-9517-b013acc6ae5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h79ks"
Jul 01 16:58:04.833263 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.833145    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7ca1dd82-196f-42ef-ac72-d0a1d9f7551f-konnectivity-ca\") pod \"konnectivity-agent-fg79l\" (UID: \"7ca1dd82-196f-42ef-ac72-d0a1d9f7551f\") " pod="kube-system/konnectivity-agent-fg79l"
Jul 01 16:58:04.833263 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.833175    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/786cf9b6-b11a-4572-9517-b013acc6ae5c-etc-selinux\") pod \"aws-ebs-csi-driver-node-h79ks\" (UID: \"786cf9b6-b11a-4572-9517-b013acc6ae5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h79ks"
Jul 01 16:58:04.833263 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.833198    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/242fcfc2-df03-4f81-bd5c-1bf8ccf2d2d4-hosts-file\") pod \"node-resolver-5bjn2\" (UID: \"242fcfc2-df03-4f81-bd5c-1bf8ccf2d2d4\") " pod="openshift-dns/node-resolver-5bjn2"
Jul 01 16:58:04.833263 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.833227    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ab1a046b-4d82-418b-9c94-3700fa11dd87-kubelet-config\") pod \"global-pull-secret-syncer-qp9lx\" (UID: \"ab1a046b-4d82-418b-9c94-3700fa11dd87\") " pod="kube-system/global-pull-secret-syncer-qp9lx"
Jul 01 16:58:04.833263 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.833252    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8ef10105-e59e-40eb-a5b7-d6eaa844ce70-etc-tuned\") pod \"tuned-7jfww\" (UID: \"8ef10105-e59e-40eb-a5b7-d6eaa844ce70\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfww"
Jul 01 16:58:04.834010 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.833277    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bf6178b8-7786-4321-9401-ca2ec4315089-host\") pod \"node-ca-bjw8j\" (UID: \"bf6178b8-7786-4321-9401-ca2ec4315089\") " pod="openshift-image-registry/node-ca-bjw8j"
Jul 01 16:58:04.834010 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.833305    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/786cf9b6-b11a-4572-9517-b013acc6ae5c-registration-dir\") pod \"aws-ebs-csi-driver-node-h79ks\" (UID: \"786cf9b6-b11a-4572-9517-b013acc6ae5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h79ks"
Jul 01 16:58:04.834010 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.833330    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/242fcfc2-df03-4f81-bd5c-1bf8ccf2d2d4-tmp-dir\") pod \"node-resolver-5bjn2\" (UID: \"242fcfc2-df03-4f81-bd5c-1bf8ccf2d2d4\") " pod="openshift-dns/node-resolver-5bjn2"
Jul 01 16:58:04.834010 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.833374    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ab1a046b-4d82-418b-9c94-3700fa11dd87-dbus\") pod \"global-pull-secret-syncer-qp9lx\" (UID: \"ab1a046b-4d82-418b-9c94-3700fa11dd87\") " pod="kube-system/global-pull-secret-syncer-qp9lx"
Jul 01 16:58:04.834010 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.833398    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/786cf9b6-b11a-4572-9517-b013acc6ae5c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-h79ks\" (UID: \"786cf9b6-b11a-4572-9517-b013acc6ae5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h79ks"
Jul 01 16:58:04.834010 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.833502    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/786cf9b6-b11a-4572-9517-b013acc6ae5c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-h79ks\" (UID: \"786cf9b6-b11a-4572-9517-b013acc6ae5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h79ks"
Jul 01 16:58:04.834010 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.833562    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8ef10105-e59e-40eb-a5b7-d6eaa844ce70-sys\") pod \"tuned-7jfww\" (UID: \"8ef10105-e59e-40eb-a5b7-d6eaa844ce70\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfww"
Jul 01 16:58:04.834010 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.833610    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ef10105-e59e-40eb-a5b7-d6eaa844ce70-var-lib-kubelet\") pod \"tuned-7jfww\" (UID: \"8ef10105-e59e-40eb-a5b7-d6eaa844ce70\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfww"
Jul 01 16:58:04.834010 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.833716    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/786cf9b6-b11a-4572-9517-b013acc6ae5c-socket-dir\") pod \"aws-ebs-csi-driver-node-h79ks\" (UID: \"786cf9b6-b11a-4572-9517-b013acc6ae5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h79ks"
Jul 01 16:58:04.834010 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.833769    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/786cf9b6-b11a-4572-9517-b013acc6ae5c-sys-fs\") pod \"aws-ebs-csi-driver-node-h79ks\" (UID: \"786cf9b6-b11a-4572-9517-b013acc6ae5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h79ks"
Jul 01 16:58:04.834010 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.833778    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bf6178b8-7786-4321-9401-ca2ec4315089-serviceca\") pod \"node-ca-bjw8j\" (UID: \"bf6178b8-7786-4321-9401-ca2ec4315089\") " pod="openshift-image-registry/node-ca-bjw8j"
Jul 01 16:58:04.834010 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.833881    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8ef10105-e59e-40eb-a5b7-d6eaa844ce70-etc-modprobe-d\") pod \"tuned-7jfww\" (UID: \"8ef10105-e59e-40eb-a5b7-d6eaa844ce70\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfww"
Jul 01 16:58:04.834010 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.833977    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8ef10105-e59e-40eb-a5b7-d6eaa844ce70-etc-systemd\") pod \"tuned-7jfww\" (UID: \"8ef10105-e59e-40eb-a5b7-d6eaa844ce70\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfww"
Jul 01 16:58:04.834010 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.834010    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8ef10105-e59e-40eb-a5b7-d6eaa844ce70-run\") pod \"tuned-7jfww\" (UID: \"8ef10105-e59e-40eb-a5b7-d6eaa844ce70\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfww"
Jul 01 16:58:04.834658 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.834075    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8ef10105-e59e-40eb-a5b7-d6eaa844ce70-lib-modules\") pod \"tuned-7jfww\" (UID: \"8ef10105-e59e-40eb-a5b7-d6eaa844ce70\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfww"
Jul 01 16:58:04.834658 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.834104    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8ef10105-e59e-40eb-a5b7-d6eaa844ce70-etc-sysconfig\") pod \"tuned-7jfww\" (UID: \"8ef10105-e59e-40eb-a5b7-d6eaa844ce70\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfww"
Jul 01 16:58:04.834658 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.834132    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ef10105-e59e-40eb-a5b7-d6eaa844ce70-etc-kubernetes\") pod \"tuned-7jfww\" (UID: \"8ef10105-e59e-40eb-a5b7-d6eaa844ce70\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfww"
Jul 01 16:58:04.834658 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.834160    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/786cf9b6-b11a-4572-9517-b013acc6ae5c-device-dir\") pod \"aws-ebs-csi-driver-node-h79ks\" (UID: \"786cf9b6-b11a-4572-9517-b013acc6ae5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h79ks"
Jul 01 16:58:04.834658 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.834625    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7ca1dd82-196f-42ef-ac72-d0a1d9f7551f-konnectivity-ca\") pod \"konnectivity-agent-fg79l\" (UID: \"7ca1dd82-196f-42ef-ac72-d0a1d9f7551f\") " pod="kube-system/konnectivity-agent-fg79l"
Jul 01 16:58:04.834896 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.834690    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bf6178b8-7786-4321-9401-ca2ec4315089-host\") pod \"node-ca-bjw8j\" (UID: \"bf6178b8-7786-4321-9401-ca2ec4315089\") " pod="openshift-image-registry/node-ca-bjw8j"
Jul 01 16:58:04.834896 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.834741    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/786cf9b6-b11a-4572-9517-b013acc6ae5c-registration-dir\") pod \"aws-ebs-csi-driver-node-h79ks\" (UID: \"786cf9b6-b11a-4572-9517-b013acc6ae5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h79ks"
Jul 01 16:58:04.834998 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.834989    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/242fcfc2-df03-4f81-bd5c-1bf8ccf2d2d4-tmp-dir\") pod \"node-resolver-5bjn2\" (UID: \"242fcfc2-df03-4f81-bd5c-1bf8ccf2d2d4\") " pod="openshift-dns/node-resolver-5bjn2"
Jul 01 16:58:04.835137 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.835121    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ab1a046b-4d82-418b-9c94-3700fa11dd87-dbus\") pod \"global-pull-secret-syncer-qp9lx\" (UID: \"ab1a046b-4d82-418b-9c94-3700fa11dd87\") " pod="kube-system/global-pull-secret-syncer-qp9lx"
Jul 01 16:58:04.835234 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.835220    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8ef10105-e59e-40eb-a5b7-d6eaa844ce70-etc-sysctl-d\") pod \"tuned-7jfww\" (UID: \"8ef10105-e59e-40eb-a5b7-d6eaa844ce70\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfww"
Jul 01 16:58:04.835390 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.835328    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8ef10105-e59e-40eb-a5b7-d6eaa844ce70-etc-sysctl-conf\") pod \"tuned-7jfww\" (UID: \"8ef10105-e59e-40eb-a5b7-d6eaa844ce70\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfww"
Jul 01 16:58:04.835464 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:04.835419    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 16:58:04.835519 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:04.835481    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab1a046b-4d82-418b-9c94-3700fa11dd87-original-pull-secret podName:ab1a046b-4d82-418b-9c94-3700fa11dd87 nodeName:}" failed. No retries permitted until 2026-07-01 16:58:05.335463503 +0000 UTC m=+3.132601898 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ab1a046b-4d82-418b-9c94-3700fa11dd87-original-pull-secret") pod "global-pull-secret-syncer-qp9lx" (UID: "ab1a046b-4d82-418b-9c94-3700fa11dd87") : object "kube-system"/"original-pull-secret" not registered
Jul 01 16:58:04.835785 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.835669    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8ef10105-e59e-40eb-a5b7-d6eaa844ce70-host\") pod \"tuned-7jfww\" (UID: \"8ef10105-e59e-40eb-a5b7-d6eaa844ce70\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfww"
Jul 01 16:58:04.835785 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.835732    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/242fcfc2-df03-4f81-bd5c-1bf8ccf2d2d4-hosts-file\") pod \"node-resolver-5bjn2\" (UID: \"242fcfc2-df03-4f81-bd5c-1bf8ccf2d2d4\") " pod="openshift-dns/node-resolver-5bjn2"
Jul 01 16:58:04.835912 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.835817    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/786cf9b6-b11a-4572-9517-b013acc6ae5c-etc-selinux\") pod \"aws-ebs-csi-driver-node-h79ks\" (UID: \"786cf9b6-b11a-4572-9517-b013acc6ae5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h79ks"
Jul 01 16:58:04.836183 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.836151    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ab1a046b-4d82-418b-9c94-3700fa11dd87-kubelet-config\") pod \"global-pull-secret-syncer-qp9lx\" (UID: \"ab1a046b-4d82-418b-9c94-3700fa11dd87\") " pod="kube-system/global-pull-secret-syncer-qp9lx"
Jul 01 16:58:04.836460 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.836432    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7ca1dd82-196f-42ef-ac72-d0a1d9f7551f-agent-certs\") pod \"konnectivity-agent-fg79l\" (UID: \"7ca1dd82-196f-42ef-ac72-d0a1d9f7551f\") " pod="kube-system/konnectivity-agent-fg79l"
Jul 01 16:58:04.837399 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.837377    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8ef10105-e59e-40eb-a5b7-d6eaa844ce70-etc-tuned\") pod \"tuned-7jfww\" (UID: \"8ef10105-e59e-40eb-a5b7-d6eaa844ce70\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfww"
Jul 01 16:58:04.838233 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.838191    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ef10105-e59e-40eb-a5b7-d6eaa844ce70-tmp\") pod \"tuned-7jfww\" (UID: \"8ef10105-e59e-40eb-a5b7-d6eaa844ce70\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfww"
Jul 01 16:58:04.849761 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.849731    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbncl\" (UniqueName: \"kubernetes.io/projected/8ef10105-e59e-40eb-a5b7-d6eaa844ce70-kube-api-access-tbncl\") pod \"tuned-7jfww\" (UID: \"8ef10105-e59e-40eb-a5b7-d6eaa844ce70\") " pod="openshift-cluster-node-tuning-operator/tuned-7jfww"
Jul 01 16:58:04.850378 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.850260    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x42d9\" (UniqueName: \"kubernetes.io/projected/242fcfc2-df03-4f81-bd5c-1bf8ccf2d2d4-kube-api-access-x42d9\") pod \"node-resolver-5bjn2\" (UID: \"242fcfc2-df03-4f81-bd5c-1bf8ccf2d2d4\") " pod="openshift-dns/node-resolver-5bjn2"
Jul 01 16:58:04.850378 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.850328    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z52b2\" (UniqueName: \"kubernetes.io/projected/786cf9b6-b11a-4572-9517-b013acc6ae5c-kube-api-access-z52b2\") pod \"aws-ebs-csi-driver-node-h79ks\" (UID: \"786cf9b6-b11a-4572-9517-b013acc6ae5c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h79ks"
Jul 01 16:58:04.850987 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.850964    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g78k\" (UniqueName: \"kubernetes.io/projected/bf6178b8-7786-4321-9401-ca2ec4315089-kube-api-access-9g78k\") pod \"node-ca-bjw8j\" (UID: \"bf6178b8-7786-4321-9401-ca2ec4315089\") " pod="openshift-image-registry/node-ca-bjw8j"
Jul 01 16:58:04.924205 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.924130    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-j6kft"
Jul 01 16:58:04.936078 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.935857    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fdj7p"
Jul 01 16:58:04.945662 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.945638    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-st8xw"
Jul 01 16:58:04.951675 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.951512    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gh58z"
Jul 01 16:58:04.958321 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.958294    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fg79l"
Jul 01 16:58:04.966131 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.966110    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h79ks"
Jul 01 16:58:04.976396 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.976221    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7jfww"
Jul 01 16:58:04.983689 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.983664    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5bjn2"
Jul 01 16:58:04.989741 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:04.989723    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bjw8j"
Jul 01 16:58:05.236959 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:05.236715    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/899ca2bf-8e98-4def-a2f9-ace75d3170a0-metrics-certs\") pod \"network-metrics-daemon-fp8j8\" (UID: \"899ca2bf-8e98-4def-a2f9-ace75d3170a0\") " pod="openshift-multus/network-metrics-daemon-fp8j8"
Jul 01 16:58:05.236959 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:05.236885    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 16:58:05.237141 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:05.236993    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/899ca2bf-8e98-4def-a2f9-ace75d3170a0-metrics-certs podName:899ca2bf-8e98-4def-a2f9-ace75d3170a0 nodeName:}" failed. No retries permitted until 2026-07-01 16:58:06.236975448 +0000 UTC m=+4.034113827 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/899ca2bf-8e98-4def-a2f9-ace75d3170a0-metrics-certs") pod "network-metrics-daemon-fp8j8" (UID: "899ca2bf-8e98-4def-a2f9-ace75d3170a0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 16:58:05.337927 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:05.337889    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab1a046b-4d82-418b-9c94-3700fa11dd87-original-pull-secret\") pod \"global-pull-secret-syncer-qp9lx\" (UID: \"ab1a046b-4d82-418b-9c94-3700fa11dd87\") " pod="kube-system/global-pull-secret-syncer-qp9lx"
Jul 01 16:58:05.338103 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:05.337943    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkwck\" (UniqueName: \"kubernetes.io/projected/d4b74cf0-0931-4466-bc62-5b807071e600-kube-api-access-bkwck\") pod \"network-check-target-txjpp\" (UID: \"d4b74cf0-0931-4466-bc62-5b807071e600\") " pod="openshift-network-diagnostics/network-check-target-txjpp"
Jul 01 16:58:05.338103 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:05.338057    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 16:58:05.338103 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:05.338078    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 16:58:05.338103 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:05.338096    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 16:58:05.338103 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:05.338108    2563 projected.go:196] Error preparing data for projected volume kube-api-access-bkwck for pod openshift-network-diagnostics/network-check-target-txjpp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 16:58:05.338276 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:05.338144    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab1a046b-4d82-418b-9c94-3700fa11dd87-original-pull-secret podName:ab1a046b-4d82-418b-9c94-3700fa11dd87 nodeName:}" failed. No retries permitted until 2026-07-01 16:58:06.338121606 +0000 UTC m=+4.135259985 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ab1a046b-4d82-418b-9c94-3700fa11dd87-original-pull-secret") pod "global-pull-secret-syncer-qp9lx" (UID: "ab1a046b-4d82-418b-9c94-3700fa11dd87") : object "kube-system"/"original-pull-secret" not registered
Jul 01 16:58:05.338276 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:05.338161    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d4b74cf0-0931-4466-bc62-5b807071e600-kube-api-access-bkwck podName:d4b74cf0-0931-4466-bc62-5b807071e600 nodeName:}" failed. No retries permitted until 2026-07-01 16:58:06.33815411 +0000 UTC m=+4.135292488 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-bkwck" (UniqueName: "kubernetes.io/projected/d4b74cf0-0931-4466-bc62-5b807071e600-kube-api-access-bkwck") pod "network-check-target-txjpp" (UID: "d4b74cf0-0931-4466-bc62-5b807071e600") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 16:58:05.674599 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:05.674518    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-06-30 16:53:04 +0000 UTC" deadline="2028-03-24 01:51:27.348387426 +0000 UTC"
Jul 01 16:58:05.674599 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:05.674545    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15152h53m21.673845455s"
Jul 01 16:58:05.790096 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:05.790066    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod242fcfc2_df03_4f81_bd5c_1bf8ccf2d2d4.slice/crio-2f73a5cfa47dc7bdca24e945bfa5f2c909c7052c3b9ad6c095ed6dbedef3e779 WatchSource:0}: Error finding container 2f73a5cfa47dc7bdca24e945bfa5f2c909c7052c3b9ad6c095ed6dbedef3e779: Status 404 returned error can't find the container with id 2f73a5cfa47dc7bdca24e945bfa5f2c909c7052c3b9ad6c095ed6dbedef3e779
Jul 01 16:58:05.812198 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:05.812169    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ca1dd82_196f_42ef_ac72_d0a1d9f7551f.slice/crio-3a1c9ded756918783b0a62f969ce7891b597691de79a7bb88cb9a9ba886e9997 WatchSource:0}: Error finding container 3a1c9ded756918783b0a62f969ce7891b597691de79a7bb88cb9a9ba886e9997: Status 404 returned error can't find the container with id 3a1c9ded756918783b0a62f969ce7891b597691de79a7bb88cb9a9ba886e9997
Jul 01 16:58:05.835764 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:05.835722    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ab87fa2_edd2_4c8b_a666_1efa2c3bae8e.slice/crio-7d20d601874e7e31e65934a31a7451400ec06256f5408a52b171122509446e57 WatchSource:0}: Error finding container 7d20d601874e7e31e65934a31a7451400ec06256f5408a52b171122509446e57: Status 404 returned error can't find the container with id 7d20d601874e7e31e65934a31a7451400ec06256f5408a52b171122509446e57
Jul 01 16:58:05.837383 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:05.837295    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ef10105_e59e_40eb_a5b7_d6eaa844ce70.slice/crio-8e74e65f3946a1380b8ba9fb7245a3b6a96ee8a3dae964cab63f0b5d58177f97 WatchSource:0}: Error finding container 8e74e65f3946a1380b8ba9fb7245a3b6a96ee8a3dae964cab63f0b5d58177f97: Status 404 returned error can't find the container with id 8e74e65f3946a1380b8ba9fb7245a3b6a96ee8a3dae964cab63f0b5d58177f97
Jul 01 16:58:05.838277 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:05.838260    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod786cf9b6_b11a_4572_9517_b013acc6ae5c.slice/crio-dcced3ad80ff079c512529e7513d7202107eb7bf4e898e84bc8fbfdcc8c00aa1 WatchSource:0}: Error finding container dcced3ad80ff079c512529e7513d7202107eb7bf4e898e84bc8fbfdcc8c00aa1: Status 404 returned error can't find the container with id dcced3ad80ff079c512529e7513d7202107eb7bf4e898e84bc8fbfdcc8c00aa1
Jul 01 16:58:05.839498 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:05.839479    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce4b7fc4_39cb_419a_aae4_30587a901329.slice/crio-44cca6275818bb9978b32c9713ee56974c8b1488e5d9ef83c580cb59e39f3a7a WatchSource:0}: Error finding container 44cca6275818bb9978b32c9713ee56974c8b1488e5d9ef83c580cb59e39f3a7a: Status 404 returned error can't find the container with id 44cca6275818bb9978b32c9713ee56974c8b1488e5d9ef83c580cb59e39f3a7a
Jul 01 16:58:05.842899 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:05.842227    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f94893d_4fd4_4745_9f71_b89dad4d1582.slice/crio-75b68b561ad18408df2deb09f53a95bf5443d77cfbe056e86129727d550640e9 WatchSource:0}: Error finding container 75b68b561ad18408df2deb09f53a95bf5443d77cfbe056e86129727d550640e9: Status 404 returned error can't find the container with id 75b68b561ad18408df2deb09f53a95bf5443d77cfbe056e86129727d550640e9
Jul 01 16:58:05.843217 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:05.843039    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf6178b8_7786_4321_9401_ca2ec4315089.slice/crio-094e2509f5963b9ceaf5b6dcba1ef956aee0af6145fa34a9b80eaa2dab485ac5 WatchSource:0}: Error finding container 094e2509f5963b9ceaf5b6dcba1ef956aee0af6145fa34a9b80eaa2dab485ac5: Status 404 returned error can't find the container with id 094e2509f5963b9ceaf5b6dcba1ef956aee0af6145fa34a9b80eaa2dab485ac5
Jul 01 16:58:05.844395 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:05.844266    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod143b1a41_fbb7_467c_8854_ba67eeb0ea3b.slice/crio-68e71630505ff35bc1c34d81dc11750048eed6e23bd184149cc87b973993094c WatchSource:0}: Error finding container 68e71630505ff35bc1c34d81dc11750048eed6e23bd184149cc87b973993094c: Status 404 returned error can't find the container with id 68e71630505ff35bc1c34d81dc11750048eed6e23bd184149cc87b973993094c
Jul 01 16:58:06.244716 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:06.244630    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/899ca2bf-8e98-4def-a2f9-ace75d3170a0-metrics-certs\") pod \"network-metrics-daemon-fp8j8\" (UID: \"899ca2bf-8e98-4def-a2f9-ace75d3170a0\") " pod="openshift-multus/network-metrics-daemon-fp8j8"
Jul 01 16:58:06.244871 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:06.244773    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 16:58:06.244871 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:06.244827    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/899ca2bf-8e98-4def-a2f9-ace75d3170a0-metrics-certs podName:899ca2bf-8e98-4def-a2f9-ace75d3170a0 nodeName:}" failed. No retries permitted until 2026-07-01 16:58:08.244813178 +0000 UTC m=+6.041951557 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/899ca2bf-8e98-4def-a2f9-ace75d3170a0-metrics-certs") pod "network-metrics-daemon-fp8j8" (UID: "899ca2bf-8e98-4def-a2f9-ace75d3170a0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 16:58:06.347359 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:06.346532    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab1a046b-4d82-418b-9c94-3700fa11dd87-original-pull-secret\") pod \"global-pull-secret-syncer-qp9lx\" (UID: \"ab1a046b-4d82-418b-9c94-3700fa11dd87\") " pod="kube-system/global-pull-secret-syncer-qp9lx"
Jul 01 16:58:06.347359 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:06.346589    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkwck\" (UniqueName: \"kubernetes.io/projected/d4b74cf0-0931-4466-bc62-5b807071e600-kube-api-access-bkwck\") pod \"network-check-target-txjpp\" (UID: \"d4b74cf0-0931-4466-bc62-5b807071e600\") " pod="openshift-network-diagnostics/network-check-target-txjpp"
Jul 01 16:58:06.347359 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:06.346769    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 16:58:06.347359 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:06.346788    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 16:58:06.347359 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:06.346800    2563 projected.go:196] Error preparing data for projected volume kube-api-access-bkwck for pod openshift-network-diagnostics/network-check-target-txjpp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 16:58:06.347359 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:06.346862    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d4b74cf0-0931-4466-bc62-5b807071e600-kube-api-access-bkwck podName:d4b74cf0-0931-4466-bc62-5b807071e600 nodeName:}" failed. No retries permitted until 2026-07-01 16:58:08.346844111 +0000 UTC m=+6.143982497 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-bkwck" (UniqueName: "kubernetes.io/projected/d4b74cf0-0931-4466-bc62-5b807071e600-kube-api-access-bkwck") pod "network-check-target-txjpp" (UID: "d4b74cf0-0931-4466-bc62-5b807071e600") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 16:58:06.347359 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:06.347254    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 16:58:06.347359 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:06.347308    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab1a046b-4d82-418b-9c94-3700fa11dd87-original-pull-secret podName:ab1a046b-4d82-418b-9c94-3700fa11dd87 nodeName:}" failed. No retries permitted until 2026-07-01 16:58:08.347293077 +0000 UTC m=+6.144431458 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ab1a046b-4d82-418b-9c94-3700fa11dd87-original-pull-secret") pod "global-pull-secret-syncer-qp9lx" (UID: "ab1a046b-4d82-418b-9c94-3700fa11dd87") : object "kube-system"/"original-pull-secret" not registered
Jul 01 16:58:06.675097 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:06.675055    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-06-30 16:53:04 +0000 UTC" deadline="2028-03-19 13:55:31.699276092 +0000 UTC"
Jul 01 16:58:06.675097 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:06.675092    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15044h57m25.024187665s"
Jul 01 16:58:06.700165 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:06.699154    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qp9lx"
Jul 01 16:58:06.700165 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:06.699266    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qp9lx" podUID="ab1a046b-4d82-418b-9c94-3700fa11dd87"
Jul 01 16:58:06.700165 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:06.699408    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fp8j8"
Jul 01 16:58:06.700165 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:06.699497    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fp8j8" podUID="899ca2bf-8e98-4def-a2f9-ace75d3170a0"
Jul 01 16:58:06.700165 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:06.699562    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-txjpp"
Jul 01 16:58:06.700165 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:06.699631    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-txjpp" podUID="d4b74cf0-0931-4466-bc62-5b807071e600"
Jul 01 16:58:06.707233 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:06.707161    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bjw8j" event={"ID":"bf6178b8-7786-4321-9401-ca2ec4315089","Type":"ContainerStarted","Data":"094e2509f5963b9ceaf5b6dcba1ef956aee0af6145fa34a9b80eaa2dab485ac5"}
Jul 01 16:58:06.714086 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:06.713988    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7jfww" event={"ID":"8ef10105-e59e-40eb-a5b7-d6eaa844ce70","Type":"ContainerStarted","Data":"8e74e65f3946a1380b8ba9fb7245a3b6a96ee8a3dae964cab63f0b5d58177f97"}
Jul 01 16:58:06.720978 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:06.720942    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fg79l" event={"ID":"7ca1dd82-196f-42ef-ac72-d0a1d9f7551f","Type":"ContainerStarted","Data":"3a1c9ded756918783b0a62f969ce7891b597691de79a7bb88cb9a9ba886e9997"}
Jul 01 16:58:06.731923 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:06.731894    2563 generic.go:356] "Generic (PLEG): container finished" podID="83e0fb78b992c4b459a7080a59353cae" containerID="0e51e364628e6d4c2c41616f6b57db91858e9feb6d7c03d61ddeb699a27977e8" exitCode=0
Jul 01 16:58:06.732447 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:06.732223    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-180.ec2.internal" event={"ID":"83e0fb78b992c4b459a7080a59353cae","Type":"ContainerDied","Data":"0e51e364628e6d4c2c41616f6b57db91858e9feb6d7c03d61ddeb699a27977e8"}
Jul 01 16:58:06.742451 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:06.742420    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-180.ec2.internal" event={"ID":"4125034d4754060c768adbf37d115970","Type":"ContainerStarted","Data":"750010ca5e5b6910d43ed8c3b04ba98ca8f1167d0131066e650043876f660127"}
Jul 01 16:58:06.758826 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:06.758798    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j6kft" event={"ID":"143b1a41-fbb7-467c-8854-ba67eeb0ea3b","Type":"ContainerStarted","Data":"68e71630505ff35bc1c34d81dc11750048eed6e23bd184149cc87b973993094c"}
Jul 01 16:58:06.761368 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:06.761289    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fdj7p" event={"ID":"6f94893d-4fd4-4745-9f71-b89dad4d1582","Type":"ContainerStarted","Data":"75b68b561ad18408df2deb09f53a95bf5443d77cfbe056e86129727d550640e9"}
Jul 01 16:58:06.765130 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:06.765104    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-st8xw" event={"ID":"ce4b7fc4-39cb-419a-aae4-30587a901329","Type":"ContainerStarted","Data":"44cca6275818bb9978b32c9713ee56974c8b1488e5d9ef83c580cb59e39f3a7a"}
Jul 01 16:58:06.775492 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:06.775437    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h79ks" event={"ID":"786cf9b6-b11a-4572-9517-b013acc6ae5c","Type":"ContainerStarted","Data":"dcced3ad80ff079c512529e7513d7202107eb7bf4e898e84bc8fbfdcc8c00aa1"}
Jul 01 16:58:06.784587 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:06.781978    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gh58z" event={"ID":"0ab87fa2-edd2-4c8b-a666-1efa2c3bae8e","Type":"ContainerStarted","Data":"7d20d601874e7e31e65934a31a7451400ec06256f5408a52b171122509446e57"}
Jul 01 16:58:06.792161 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:06.792114    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5bjn2" event={"ID":"242fcfc2-df03-4f81-bd5c-1bf8ccf2d2d4","Type":"ContainerStarted","Data":"2f73a5cfa47dc7bdca24e945bfa5f2c909c7052c3b9ad6c095ed6dbedef3e779"}
Jul 01 16:58:07.808636 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:07.808586    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-180.ec2.internal" event={"ID":"83e0fb78b992c4b459a7080a59353cae","Type":"ContainerStarted","Data":"2773bfd1501f757c47fd165e4b73f8410e36f7d0be76a7271d77a480ae6e1ba3"}
Jul 01 16:58:07.825618 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:07.824939    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-180.ec2.internal" podStartSLOduration=3.824908643 podStartE2EDuration="3.824908643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 16:58:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 16:58:06.769504759 +0000 UTC m=+4.566643182" watchObservedRunningTime="2026-07-01 16:58:07.824908643 +0000 UTC m=+5.622047044"
Jul 01 16:58:08.266955 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:08.266869    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/899ca2bf-8e98-4def-a2f9-ace75d3170a0-metrics-certs\") pod \"network-metrics-daemon-fp8j8\" (UID: \"899ca2bf-8e98-4def-a2f9-ace75d3170a0\") " pod="openshift-multus/network-metrics-daemon-fp8j8"
Jul 01 16:58:08.267124 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:08.267017    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 16:58:08.267124 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:08.267079    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/899ca2bf-8e98-4def-a2f9-ace75d3170a0-metrics-certs podName:899ca2bf-8e98-4def-a2f9-ace75d3170a0 nodeName:}" failed. No retries permitted until 2026-07-01 16:58:12.267060172 +0000 UTC m=+10.064198568 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/899ca2bf-8e98-4def-a2f9-ace75d3170a0-metrics-certs") pod "network-metrics-daemon-fp8j8" (UID: "899ca2bf-8e98-4def-a2f9-ace75d3170a0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 16:58:08.368443 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:08.367586    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab1a046b-4d82-418b-9c94-3700fa11dd87-original-pull-secret\") pod \"global-pull-secret-syncer-qp9lx\" (UID: \"ab1a046b-4d82-418b-9c94-3700fa11dd87\") " pod="kube-system/global-pull-secret-syncer-qp9lx"
Jul 01 16:58:08.368443 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:08.367646    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkwck\" (UniqueName: \"kubernetes.io/projected/d4b74cf0-0931-4466-bc62-5b807071e600-kube-api-access-bkwck\") pod \"network-check-target-txjpp\" (UID: \"d4b74cf0-0931-4466-bc62-5b807071e600\") " pod="openshift-network-diagnostics/network-check-target-txjpp"
Jul 01 16:58:08.368443 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:08.367821    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 16:58:08.368443 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:08.367842    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 16:58:08.368443 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:08.367863    2563 projected.go:196] Error preparing data for projected volume kube-api-access-bkwck for pod openshift-network-diagnostics/network-check-target-txjpp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 16:58:08.368443 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:08.367920    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d4b74cf0-0931-4466-bc62-5b807071e600-kube-api-access-bkwck podName:d4b74cf0-0931-4466-bc62-5b807071e600 nodeName:}" failed. No retries permitted until 2026-07-01 16:58:12.367903291 +0000 UTC m=+10.165041673 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-bkwck" (UniqueName: "kubernetes.io/projected/d4b74cf0-0931-4466-bc62-5b807071e600-kube-api-access-bkwck") pod "network-check-target-txjpp" (UID: "d4b74cf0-0931-4466-bc62-5b807071e600") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 16:58:08.368443 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:08.368331    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 16:58:08.368443 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:08.368396    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab1a046b-4d82-418b-9c94-3700fa11dd87-original-pull-secret podName:ab1a046b-4d82-418b-9c94-3700fa11dd87 nodeName:}" failed. No retries permitted until 2026-07-01 16:58:12.368380565 +0000 UTC m=+10.165518951 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ab1a046b-4d82-418b-9c94-3700fa11dd87-original-pull-secret") pod "global-pull-secret-syncer-qp9lx" (UID: "ab1a046b-4d82-418b-9c94-3700fa11dd87") : object "kube-system"/"original-pull-secret" not registered
Jul 01 16:58:08.695927 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:08.695852    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qp9lx"
Jul 01 16:58:08.696203 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:08.696014    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qp9lx" podUID="ab1a046b-4d82-418b-9c94-3700fa11dd87"
Jul 01 16:58:08.696516 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:08.696481    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fp8j8"
Jul 01 16:58:08.696613 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:08.696586    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fp8j8" podUID="899ca2bf-8e98-4def-a2f9-ace75d3170a0"
Jul 01 16:58:08.696681 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:08.696644    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-txjpp"
Jul 01 16:58:08.696739 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:08.696711    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-txjpp" podUID="d4b74cf0-0931-4466-bc62-5b807071e600"
Jul 01 16:58:10.691858 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:10.691821    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qp9lx"
Jul 01 16:58:10.692308 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:10.691982    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qp9lx" podUID="ab1a046b-4d82-418b-9c94-3700fa11dd87"
Jul 01 16:58:10.692308 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:10.692285    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fp8j8"
Jul 01 16:58:10.692444 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:10.692379    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fp8j8" podUID="899ca2bf-8e98-4def-a2f9-ace75d3170a0"
Jul 01 16:58:10.692444 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:10.692419    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-txjpp"
Jul 01 16:58:10.692548 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:10.692506    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-txjpp" podUID="d4b74cf0-0931-4466-bc62-5b807071e600"
Jul 01 16:58:12.301749 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:12.301711    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/899ca2bf-8e98-4def-a2f9-ace75d3170a0-metrics-certs\") pod \"network-metrics-daemon-fp8j8\" (UID: \"899ca2bf-8e98-4def-a2f9-ace75d3170a0\") " pod="openshift-multus/network-metrics-daemon-fp8j8"
Jul 01 16:58:12.302203 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:12.301847    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 16:58:12.302203 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:12.301907    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/899ca2bf-8e98-4def-a2f9-ace75d3170a0-metrics-certs podName:899ca2bf-8e98-4def-a2f9-ace75d3170a0 nodeName:}" failed. No retries permitted until 2026-07-01 16:58:20.301886657 +0000 UTC m=+18.099025053 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/899ca2bf-8e98-4def-a2f9-ace75d3170a0-metrics-certs") pod "network-metrics-daemon-fp8j8" (UID: "899ca2bf-8e98-4def-a2f9-ace75d3170a0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 16:58:12.402776 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:12.402735    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab1a046b-4d82-418b-9c94-3700fa11dd87-original-pull-secret\") pod \"global-pull-secret-syncer-qp9lx\" (UID: \"ab1a046b-4d82-418b-9c94-3700fa11dd87\") " pod="kube-system/global-pull-secret-syncer-qp9lx"
Jul 01 16:58:12.402949 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:12.402789    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkwck\" (UniqueName: \"kubernetes.io/projected/d4b74cf0-0931-4466-bc62-5b807071e600-kube-api-access-bkwck\") pod \"network-check-target-txjpp\" (UID: \"d4b74cf0-0931-4466-bc62-5b807071e600\") " pod="openshift-network-diagnostics/network-check-target-txjpp"
Jul 01 16:58:12.403026 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:12.402949    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 16:58:12.403026 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:12.402968    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 16:58:12.403026 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:12.402981    2563 projected.go:196] Error preparing data for projected volume kube-api-access-bkwck for pod openshift-network-diagnostics/network-check-target-txjpp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 16:58:12.403179 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:12.403034    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d4b74cf0-0931-4466-bc62-5b807071e600-kube-api-access-bkwck podName:d4b74cf0-0931-4466-bc62-5b807071e600 nodeName:}" failed. No retries permitted until 2026-07-01 16:58:20.403016997 +0000 UTC m=+18.200155381 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-bkwck" (UniqueName: "kubernetes.io/projected/d4b74cf0-0931-4466-bc62-5b807071e600-kube-api-access-bkwck") pod "network-check-target-txjpp" (UID: "d4b74cf0-0931-4466-bc62-5b807071e600") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 16:58:12.403433 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:12.403414    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 16:58:12.403511 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:12.403468    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab1a046b-4d82-418b-9c94-3700fa11dd87-original-pull-secret podName:ab1a046b-4d82-418b-9c94-3700fa11dd87 nodeName:}" failed. No retries permitted until 2026-07-01 16:58:20.403453034 +0000 UTC m=+18.200591430 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ab1a046b-4d82-418b-9c94-3700fa11dd87-original-pull-secret") pod "global-pull-secret-syncer-qp9lx" (UID: "ab1a046b-4d82-418b-9c94-3700fa11dd87") : object "kube-system"/"original-pull-secret" not registered
Jul 01 16:58:12.692697 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:12.692619    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-txjpp"
Jul 01 16:58:12.692846 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:12.692741    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-txjpp" podUID="d4b74cf0-0931-4466-bc62-5b807071e600"
Jul 01 16:58:12.692916 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:12.692899    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qp9lx"
Jul 01 16:58:12.693008 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:12.692984    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qp9lx" podUID="ab1a046b-4d82-418b-9c94-3700fa11dd87"
Jul 01 16:58:12.693058 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:12.693045    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fp8j8"
Jul 01 16:58:12.693127 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:12.693103    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fp8j8" podUID="899ca2bf-8e98-4def-a2f9-ace75d3170a0"
Jul 01 16:58:14.691699 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:14.691620    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-txjpp"
Jul 01 16:58:14.692135 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:14.691635    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fp8j8"
Jul 01 16:58:14.692135 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:14.691738    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qp9lx"
Jul 01 16:58:14.692135 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:14.691747    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-txjpp" podUID="d4b74cf0-0931-4466-bc62-5b807071e600"
Jul 01 16:58:14.692135 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:14.691843    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fp8j8" podUID="899ca2bf-8e98-4def-a2f9-ace75d3170a0"
Jul 01 16:58:14.692135 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:14.691905    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qp9lx" podUID="ab1a046b-4d82-418b-9c94-3700fa11dd87"
Jul 01 16:58:16.691719 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:16.691676    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-txjpp"
Jul 01 16:58:16.692152 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:16.691676    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qp9lx"
Jul 01 16:58:16.692152 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:16.691806    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-txjpp" podUID="d4b74cf0-0931-4466-bc62-5b807071e600"
Jul 01 16:58:16.692152 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:16.691890    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qp9lx" podUID="ab1a046b-4d82-418b-9c94-3700fa11dd87"
Jul 01 16:58:16.692152 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:16.691689    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fp8j8"
Jul 01 16:58:16.692152 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:16.692005    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fp8j8" podUID="899ca2bf-8e98-4def-a2f9-ace75d3170a0"
Jul 01 16:58:18.691744 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:18.691365    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-txjpp"
Jul 01 16:58:18.691744 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:18.691370    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fp8j8"
Jul 01 16:58:18.691744 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:18.691485    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qp9lx"
Jul 01 16:58:18.691744 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:18.691489    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-txjpp" podUID="d4b74cf0-0931-4466-bc62-5b807071e600"
Jul 01 16:58:18.691744 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:18.691606    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fp8j8" podUID="899ca2bf-8e98-4def-a2f9-ace75d3170a0"
Jul 01 16:58:18.691744 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:18.691684    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qp9lx" podUID="ab1a046b-4d82-418b-9c94-3700fa11dd87"
Jul 01 16:58:20.366865 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:20.366830    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/899ca2bf-8e98-4def-a2f9-ace75d3170a0-metrics-certs\") pod \"network-metrics-daemon-fp8j8\" (UID: \"899ca2bf-8e98-4def-a2f9-ace75d3170a0\") " pod="openshift-multus/network-metrics-daemon-fp8j8"
Jul 01 16:58:20.367468 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:20.366951    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 16:58:20.367468 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:20.367029    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/899ca2bf-8e98-4def-a2f9-ace75d3170a0-metrics-certs podName:899ca2bf-8e98-4def-a2f9-ace75d3170a0 nodeName:}" failed. No retries permitted until 2026-07-01 16:58:36.367013652 +0000 UTC m=+34.164152047 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/899ca2bf-8e98-4def-a2f9-ace75d3170a0-metrics-certs") pod "network-metrics-daemon-fp8j8" (UID: "899ca2bf-8e98-4def-a2f9-ace75d3170a0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 16:58:20.467776 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:20.467747    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab1a046b-4d82-418b-9c94-3700fa11dd87-original-pull-secret\") pod \"global-pull-secret-syncer-qp9lx\" (UID: \"ab1a046b-4d82-418b-9c94-3700fa11dd87\") " pod="kube-system/global-pull-secret-syncer-qp9lx"
Jul 01 16:58:20.467897 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:20.467787    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkwck\" (UniqueName: \"kubernetes.io/projected/d4b74cf0-0931-4466-bc62-5b807071e600-kube-api-access-bkwck\") pod \"network-check-target-txjpp\" (UID: \"d4b74cf0-0931-4466-bc62-5b807071e600\") " pod="openshift-network-diagnostics/network-check-target-txjpp"
Jul 01 16:58:20.467897 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:20.467888    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 16:58:20.467962 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:20.467911    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 16:58:20.467962 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:20.467925    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 16:58:20.467962 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:20.467934    2563 projected.go:196] Error preparing data for projected volume kube-api-access-bkwck for pod openshift-network-diagnostics/network-check-target-txjpp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 16:58:20.467962 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:20.467946    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab1a046b-4d82-418b-9c94-3700fa11dd87-original-pull-secret podName:ab1a046b-4d82-418b-9c94-3700fa11dd87 nodeName:}" failed. No retries permitted until 2026-07-01 16:58:36.467930989 +0000 UTC m=+34.265069368 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ab1a046b-4d82-418b-9c94-3700fa11dd87-original-pull-secret") pod "global-pull-secret-syncer-qp9lx" (UID: "ab1a046b-4d82-418b-9c94-3700fa11dd87") : object "kube-system"/"original-pull-secret" not registered
Jul 01 16:58:20.468085 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:20.467976    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d4b74cf0-0931-4466-bc62-5b807071e600-kube-api-access-bkwck podName:d4b74cf0-0931-4466-bc62-5b807071e600 nodeName:}" failed. No retries permitted until 2026-07-01 16:58:36.467960933 +0000 UTC m=+34.265099332 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-bkwck" (UniqueName: "kubernetes.io/projected/d4b74cf0-0931-4466-bc62-5b807071e600-kube-api-access-bkwck") pod "network-check-target-txjpp" (UID: "d4b74cf0-0931-4466-bc62-5b807071e600") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 16:58:20.692008 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:20.691925    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-txjpp"
Jul 01 16:58:20.692158 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:20.691928    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qp9lx"
Jul 01 16:58:20.692158 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:20.692047    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-txjpp" podUID="d4b74cf0-0931-4466-bc62-5b807071e600"
Jul 01 16:58:20.692158 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:20.692065    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fp8j8"
Jul 01 16:58:20.692306 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:20.692197    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qp9lx" podUID="ab1a046b-4d82-418b-9c94-3700fa11dd87"
Jul 01 16:58:20.692306 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:20.692253    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fp8j8" podUID="899ca2bf-8e98-4def-a2f9-ace75d3170a0"
Jul 01 16:58:22.692967 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:22.692743    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fp8j8"
Jul 01 16:58:22.693457 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:22.692836    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-txjpp"
Jul 01 16:58:22.693457 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:22.693053    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fp8j8" podUID="899ca2bf-8e98-4def-a2f9-ace75d3170a0"
Jul 01 16:58:22.693457 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:22.692867    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qp9lx"
Jul 01 16:58:22.693457 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:22.693132    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-txjpp" podUID="d4b74cf0-0931-4466-bc62-5b807071e600"
Jul 01 16:58:22.693457 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:22.693190    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qp9lx" podUID="ab1a046b-4d82-418b-9c94-3700fa11dd87"
Jul 01 16:58:24.691849 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:24.691816    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fp8j8"
Jul 01 16:58:24.692293 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:24.691816    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-txjpp"
Jul 01 16:58:24.692293 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:24.691961    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qp9lx"
Jul 01 16:58:24.692293 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:24.691960    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fp8j8" podUID="899ca2bf-8e98-4def-a2f9-ace75d3170a0"
Jul 01 16:58:24.692293 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:24.692047    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-txjpp" podUID="d4b74cf0-0931-4466-bc62-5b807071e600"
Jul 01 16:58:24.692293 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:24.692124    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qp9lx" podUID="ab1a046b-4d82-418b-9c94-3700fa11dd87"
Jul 01 16:58:25.849060 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:25.848853    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j6kft" event={"ID":"143b1a41-fbb7-467c-8854-ba67eeb0ea3b","Type":"ContainerStarted","Data":"9b9f278d77cc0df054d1457bd84d7cb0742f0791d17834e64e6bf387413db291"}
Jul 01 16:58:25.849834 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:25.849149    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j6kft" event={"ID":"143b1a41-fbb7-467c-8854-ba67eeb0ea3b","Type":"ContainerStarted","Data":"0c79e371ef1bb784fa0da55f913a07c01cc5f69412aa3f9c49ce989dc2ff56c5"}
Jul 01 16:58:25.849834 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:25.849202    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j6kft" event={"ID":"143b1a41-fbb7-467c-8854-ba67eeb0ea3b","Type":"ContainerStarted","Data":"7a1e77cf539f30f380e4ca4dc1ec2847a7d570fc8d0d3ae678a129a312e8dc73"}
Jul 01 16:58:25.849834 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:25.849239    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j6kft" event={"ID":"143b1a41-fbb7-467c-8854-ba67eeb0ea3b","Type":"ContainerStarted","Data":"0ed68c9bcb3dbb4c688d93e2b10c475bff8a519f90ad847f942dcdb1693952e4"}
Jul 01 16:58:25.849834 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:25.849273    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j6kft" event={"ID":"143b1a41-fbb7-467c-8854-ba67eeb0ea3b","Type":"ContainerStarted","Data":"a39dbc3af47dd9ad4d59c0e7d334b0898f015d60f1cfba309dad1af50e78de7d"}
Jul 01 16:58:25.849834 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:25.849317    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j6kft" event={"ID":"143b1a41-fbb7-467c-8854-ba67eeb0ea3b","Type":"ContainerStarted","Data":"72d685c4aa1d9482e2240d6b23dd1caa8255096e6e12fd7deae8481f23af75ff"}
Jul 01 16:58:25.850257 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:25.850237    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-st8xw" event={"ID":"ce4b7fc4-39cb-419a-aae4-30587a901329","Type":"ContainerStarted","Data":"4bcc92747b8ef5dcda4922fab8ea9dfd2dc650b2d58db4f4d7190dd78ff4e327"}
Jul 01 16:58:25.851653 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:25.851625    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h79ks" event={"ID":"786cf9b6-b11a-4572-9517-b013acc6ae5c","Type":"ContainerStarted","Data":"81702b50776978ae663d4fac21509a0652e20b4d20b917a5ee791b4b2147c469"}
Jul 01 16:58:25.853035 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:25.853015    2563 generic.go:356] "Generic (PLEG): container finished" podID="0ab87fa2-edd2-4c8b-a666-1efa2c3bae8e" containerID="ba9985cdcce12a278af3a512d2dacf7aa294f660b3702389ea568b463c101ff6" exitCode=0
Jul 01 16:58:25.853122 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:25.853092    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gh58z" event={"ID":"0ab87fa2-edd2-4c8b-a666-1efa2c3bae8e","Type":"ContainerDied","Data":"ba9985cdcce12a278af3a512d2dacf7aa294f660b3702389ea568b463c101ff6"}
Jul 01 16:58:25.854400 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:25.854378    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5bjn2" event={"ID":"242fcfc2-df03-4f81-bd5c-1bf8ccf2d2d4","Type":"ContainerStarted","Data":"1be8b55f53a1741dd98822fc6300c166415443e90158a849ed15e3a7476d545d"}
Jul 01 16:58:25.855859 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:25.855836    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bjw8j" event={"ID":"bf6178b8-7786-4321-9401-ca2ec4315089","Type":"ContainerStarted","Data":"afa686c5fe0d149d43d383fae8f72e56d9935122961d116d4fc7018e0bdfeec5"}
Jul 01 16:58:25.857388 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:25.857368    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7jfww" event={"ID":"8ef10105-e59e-40eb-a5b7-d6eaa844ce70","Type":"ContainerStarted","Data":"85ea61482bdec36c7440bb5a5688f27e7bd4615c0322061bd4e0961cec53d68b"}
Jul 01 16:58:25.858662 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:25.858645    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fg79l" event={"ID":"7ca1dd82-196f-42ef-ac72-d0a1d9f7551f","Type":"ContainerStarted","Data":"d499bea4e73ea59ede62dcbe090cca18ccfa0b2df79ec5f531ec341f915c0834"}
Jul 01 16:58:25.886920 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:25.886873    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-180.ec2.internal" podStartSLOduration=21.886860855 podStartE2EDuration="21.886860855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 16:58:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 16:58:07.825416498 +0000 UTC m=+5.622554916" watchObservedRunningTime="2026-07-01 16:58:25.886860855 +0000 UTC m=+23.683999257"
Jul 01 16:58:25.887033 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:25.886997    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-st8xw" podStartSLOduration=3.712701194 podStartE2EDuration="22.886993616s" totalImagesPullingTime="19.174292422s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 16:58:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 16:58:25.886750369 +0000 UTC m=+23.683888770" watchObservedRunningTime="2026-07-01 16:58:25.886993616 +0000 UTC m=+23.684132017"
Jul 01 16:58:25.903790 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:25.903752    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-fg79l" podStartSLOduration=3.792083902 podStartE2EDuration="22.903738832s" totalImagesPullingTime="19.11165493s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 16:58:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 16:58:25.903516751 +0000 UTC m=+23.700655153" watchObservedRunningTime="2026-07-01 16:58:25.903738832 +0000 UTC m=+23.700877233"
Jul 01 16:58:25.923278 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:25.923239    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-7jfww" podStartSLOduration=3.801533274 podStartE2EDuration="22.923226117s" totalImagesPullingTime="19.121692843s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 16:58:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 16:58:25.92284342 +0000 UTC m=+23.719981820" watchObservedRunningTime="2026-07-01 16:58:25.923226117 +0000 UTC m=+23.720364518"
Jul 01 16:58:25.944296 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:25.944254    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-5bjn2" podStartSLOduration=3.8088871319999997 podStartE2EDuration="22.944244234s" totalImagesPullingTime="19.135357102s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 16:58:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 16:58:25.943834691 +0000 UTC m=+23.740973092" watchObservedRunningTime="2026-07-01 16:58:25.944244234 +0000 UTC m=+23.741382635"
Jul 01 16:58:25.969909 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:25.969874    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-bjw8j" podStartSLOduration=8.372198823 podStartE2EDuration="22.96986143s" totalImagesPullingTime="14.597662607s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 16:58:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 16:58:25.96949945 +0000 UTC m=+23.766637847" watchObservedRunningTime="2026-07-01 16:58:25.96986143 +0000 UTC m=+23.766999830"
Jul 01 16:58:26.691370 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:26.691277    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-txjpp"
Jul 01 16:58:26.691596 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:26.691278    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qp9lx"
Jul 01 16:58:26.691596 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:26.691407    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-txjpp" podUID="d4b74cf0-0931-4466-bc62-5b807071e600"
Jul 01 16:58:26.691596 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:26.691278    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fp8j8"
Jul 01 16:58:26.691596 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:26.691466    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qp9lx" podUID="ab1a046b-4d82-418b-9c94-3700fa11dd87"
Jul 01 16:58:26.691596 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:26.691543    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fp8j8" podUID="899ca2bf-8e98-4def-a2f9-ace75d3170a0"
Jul 01 16:58:26.862708 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:26.862636    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fdj7p" event={"ID":"6f94893d-4fd4-4745-9f71-b89dad4d1582","Type":"ContainerStarted","Data":"a6782d480c2c7ef3eb18a671646fade25e90a6e72394053e189e316ad3d4e93f"}
Jul 01 16:58:26.884738 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:26.884681    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-fdj7p" podStartSLOduration=5.779853881 podStartE2EDuration="24.884662392s" totalImagesPullingTime="19.104808511s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 16:58:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 16:58:26.883624036 +0000 UTC m=+24.680762455" watchObservedRunningTime="2026-07-01 16:58:26.884662392 +0000 UTC m=+24.681800795"
Jul 01 16:58:27.294586 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:27.294328    2563 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 01 16:58:27.461904 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:27.461744    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-fg79l"
Jul 01 16:58:27.462333 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:27.462311    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-fg79l"
Jul 01 16:58:27.693161 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:27.693066    2563 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-01T16:58:27.294577348Z","UUID":"6d68ec47-2770-4f3a-b965-199d585c4db9","Handler":null,"Name":"","Endpoint":""}
Jul 01 16:58:27.696435 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:27.696411    2563 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 01 16:58:27.696587 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:27.696457    2563 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 01 16:58:27.871564 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:27.871529    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h79ks" event={"ID":"786cf9b6-b11a-4572-9517-b013acc6ae5c","Type":"ContainerStarted","Data":"84721dd80840a807c1c6c820d6f5f332561157c8ec2cefa5bb29182cd96d9682"}
Jul 01 16:58:27.872052 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:27.871921    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-fg79l"
Jul 01 16:58:27.872573 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:27.872548    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-fg79l"
Jul 01 16:58:28.691862 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:28.691834    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fp8j8"
Jul 01 16:58:28.692038 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:28.691834    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-txjpp"
Jul 01 16:58:28.692038 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:28.692022    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qp9lx"
Jul 01 16:58:28.692038 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:28.692018    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fp8j8" podUID="899ca2bf-8e98-4def-a2f9-ace75d3170a0"
Jul 01 16:58:28.692198 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:28.692114    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-txjpp" podUID="d4b74cf0-0931-4466-bc62-5b807071e600"
Jul 01 16:58:28.692198 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:28.692192    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qp9lx" podUID="ab1a046b-4d82-418b-9c94-3700fa11dd87"
Jul 01 16:58:28.877907 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:28.877799    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j6kft" event={"ID":"143b1a41-fbb7-467c-8854-ba67eeb0ea3b","Type":"ContainerStarted","Data":"783a148b287f4f680f93d8469073df7d855fbc8a0d1937d0da2c0b498154089f"}
Jul 01 16:58:29.882406 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:29.882363    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h79ks" event={"ID":"786cf9b6-b11a-4572-9517-b013acc6ae5c","Type":"ContainerStarted","Data":"e073c72d702fb4520b711676a7f38d31983ff99ab5447dac4fd6f87341e1b8ba"}
Jul 01 16:58:29.903764 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:29.903712    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h79ks" podStartSLOduration=4.308582871 podStartE2EDuration="26.903695673s" totalImagesPullingTime="22.595112802s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 16:58:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 16:58:29.90317403 +0000 UTC m=+27.700312438" watchObservedRunningTime="2026-07-01 16:58:29.903695673 +0000 UTC m=+27.700834073"
Jul 01 16:58:30.692021 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:30.691636    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fp8j8"
Jul 01 16:58:30.692021 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:30.691646    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-txjpp"
Jul 01 16:58:30.692021 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:30.691963    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fp8j8" podUID="899ca2bf-8e98-4def-a2f9-ace75d3170a0"
Jul 01 16:58:30.692542 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:30.691658    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qp9lx"
Jul 01 16:58:30.692542 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:30.692033    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-txjpp" podUID="d4b74cf0-0931-4466-bc62-5b807071e600"
Jul 01 16:58:30.692542 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:30.692136    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qp9lx" podUID="ab1a046b-4d82-418b-9c94-3700fa11dd87"
Jul 01 16:58:30.886435 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:30.886398    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gh58z" event={"ID":"0ab87fa2-edd2-4c8b-a666-1efa2c3bae8e","Type":"ContainerStarted","Data":"6b3255bf79f4e7066bd682ff4783224857fbe9760ee13839906064d9c26c3a5b"}
Jul 01 16:58:30.890410 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:30.890379    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j6kft" event={"ID":"143b1a41-fbb7-467c-8854-ba67eeb0ea3b","Type":"ContainerStarted","Data":"e884b955ab8fbb37813beb87869e7a2fdd00ab601c647c5be815ec8cb03888f4"}
Jul 01 16:58:30.943451 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:30.943407    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-j6kft" podStartSLOduration=9.774132973 podStartE2EDuration="28.943390244s" totalImagesPullingTime="19.169257271s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 16:58:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 16:58:30.943263366 +0000 UTC m=+28.740401759" watchObservedRunningTime="2026-07-01 16:58:30.943390244 +0000 UTC m=+28.740528641"
Jul 01 16:58:31.897427 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:31.895147    2563 generic.go:356] "Generic (PLEG): container finished" podID="0ab87fa2-edd2-4c8b-a666-1efa2c3bae8e" containerID="6b3255bf79f4e7066bd682ff4783224857fbe9760ee13839906064d9c26c3a5b" exitCode=0
Jul 01 16:58:31.897427 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:31.896917    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gh58z" event={"ID":"0ab87fa2-edd2-4c8b-a666-1efa2c3bae8e","Type":"ContainerDied","Data":"6b3255bf79f4e7066bd682ff4783224857fbe9760ee13839906064d9c26c3a5b"}
Jul 01 16:58:31.897427 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:31.896969    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j6kft"
Jul 01 16:58:31.897427 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:31.897084    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j6kft"
Jul 01 16:58:31.897427 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:31.897140    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j6kft"
Jul 01 16:58:31.931215 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:31.931186    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j6kft"
Jul 01 16:58:31.932533 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:31.932392    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j6kft"
Jul 01 16:58:32.696416 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:32.696381    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qp9lx"
Jul 01 16:58:32.696641 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:32.696516    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-txjpp"
Jul 01 16:58:32.696641 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:32.696512    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qp9lx" podUID="ab1a046b-4d82-418b-9c94-3700fa11dd87"
Jul 01 16:58:32.696641 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:32.696563    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fp8j8"
Jul 01 16:58:32.696783 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:32.696647    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-txjpp" podUID="d4b74cf0-0931-4466-bc62-5b807071e600"
Jul 01 16:58:32.696851 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:32.696812    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fp8j8" podUID="899ca2bf-8e98-4def-a2f9-ace75d3170a0"
Jul 01 16:58:33.105827 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:33.105572    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fp8j8"]
Jul 01 16:58:33.106241 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:33.105921    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fp8j8"
Jul 01 16:58:33.106241 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:33.106036    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fp8j8" podUID="899ca2bf-8e98-4def-a2f9-ace75d3170a0"
Jul 01 16:58:33.112683 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:33.112659    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-txjpp"]
Jul 01 16:58:33.112784 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:33.112748    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-txjpp"
Jul 01 16:58:33.112851 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:33.112826    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-txjpp" podUID="d4b74cf0-0931-4466-bc62-5b807071e600"
Jul 01 16:58:33.114867 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:33.114845    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qp9lx"]
Jul 01 16:58:33.114996 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:33.114928    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qp9lx"
Jul 01 16:58:33.115050 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:33.115016    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qp9lx" podUID="ab1a046b-4d82-418b-9c94-3700fa11dd87"
Jul 01 16:58:33.901202 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:33.901164    2563 generic.go:356] "Generic (PLEG): container finished" podID="0ab87fa2-edd2-4c8b-a666-1efa2c3bae8e" containerID="aac993be97173a867bd96c36d8d0881cfcbdc3aec502d9c7591299d0b1f1d3f6" exitCode=0
Jul 01 16:58:33.901409 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:33.901245    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gh58z" event={"ID":"0ab87fa2-edd2-4c8b-a666-1efa2c3bae8e","Type":"ContainerDied","Data":"aac993be97173a867bd96c36d8d0881cfcbdc3aec502d9c7591299d0b1f1d3f6"}
Jul 01 16:58:34.691819 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:34.691792    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-txjpp"
Jul 01 16:58:34.692432 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:34.691896    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-txjpp" podUID="d4b74cf0-0931-4466-bc62-5b807071e600"
Jul 01 16:58:34.692432 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:34.692289    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qp9lx"
Jul 01 16:58:34.692432 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:34.692382    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qp9lx" podUID="ab1a046b-4d82-418b-9c94-3700fa11dd87"
Jul 01 16:58:34.692607 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:34.692508    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fp8j8"
Jul 01 16:58:34.692659 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:34.692603    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fp8j8" podUID="899ca2bf-8e98-4def-a2f9-ace75d3170a0"
Jul 01 16:58:35.909633 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:35.909598    2563 generic.go:356] "Generic (PLEG): container finished" podID="0ab87fa2-edd2-4c8b-a666-1efa2c3bae8e" containerID="4ffbfa07dbe4a6965e174637d16262cff21046597160bec4c3e0b5eaf2042881" exitCode=0
Jul 01 16:58:35.910319 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:35.909644    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gh58z" event={"ID":"0ab87fa2-edd2-4c8b-a666-1efa2c3bae8e","Type":"ContainerDied","Data":"4ffbfa07dbe4a6965e174637d16262cff21046597160bec4c3e0b5eaf2042881"}
Jul 01 16:58:36.395283 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:36.395248    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/899ca2bf-8e98-4def-a2f9-ace75d3170a0-metrics-certs\") pod \"network-metrics-daemon-fp8j8\" (UID: \"899ca2bf-8e98-4def-a2f9-ace75d3170a0\") " pod="openshift-multus/network-metrics-daemon-fp8j8"
Jul 01 16:58:36.395501 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:36.395388    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 16:58:36.395501 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:36.395436    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/899ca2bf-8e98-4def-a2f9-ace75d3170a0-metrics-certs podName:899ca2bf-8e98-4def-a2f9-ace75d3170a0 nodeName:}" failed. No retries permitted until 2026-07-01 16:59:08.395424305 +0000 UTC m=+66.192562683 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/899ca2bf-8e98-4def-a2f9-ace75d3170a0-metrics-certs") pod "network-metrics-daemon-fp8j8" (UID: "899ca2bf-8e98-4def-a2f9-ace75d3170a0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 16:58:36.496382 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:36.496327    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab1a046b-4d82-418b-9c94-3700fa11dd87-original-pull-secret\") pod \"global-pull-secret-syncer-qp9lx\" (UID: \"ab1a046b-4d82-418b-9c94-3700fa11dd87\") " pod="kube-system/global-pull-secret-syncer-qp9lx"
Jul 01 16:58:36.496521 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:36.496403    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkwck\" (UniqueName: \"kubernetes.io/projected/d4b74cf0-0931-4466-bc62-5b807071e600-kube-api-access-bkwck\") pod \"network-check-target-txjpp\" (UID: \"d4b74cf0-0931-4466-bc62-5b807071e600\") " pod="openshift-network-diagnostics/network-check-target-txjpp"
Jul 01 16:58:36.496521 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:36.496482    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 16:58:36.496647 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:36.496547    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab1a046b-4d82-418b-9c94-3700fa11dd87-original-pull-secret podName:ab1a046b-4d82-418b-9c94-3700fa11dd87 nodeName:}" failed. No retries permitted until 2026-07-01 16:59:08.496530864 +0000 UTC m=+66.293669246 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ab1a046b-4d82-418b-9c94-3700fa11dd87-original-pull-secret") pod "global-pull-secret-syncer-qp9lx" (UID: "ab1a046b-4d82-418b-9c94-3700fa11dd87") : object "kube-system"/"original-pull-secret" not registered
Jul 01 16:58:36.496647 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:36.496572    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 16:58:36.496647 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:36.496601    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 16:58:36.496647 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:36.496614    2563 projected.go:196] Error preparing data for projected volume kube-api-access-bkwck for pod openshift-network-diagnostics/network-check-target-txjpp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 16:58:36.496794 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:36.496671    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d4b74cf0-0931-4466-bc62-5b807071e600-kube-api-access-bkwck podName:d4b74cf0-0931-4466-bc62-5b807071e600 nodeName:}" failed. No retries permitted until 2026-07-01 16:59:08.496655399 +0000 UTC m=+66.293793800 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-bkwck" (UniqueName: "kubernetes.io/projected/d4b74cf0-0931-4466-bc62-5b807071e600-kube-api-access-bkwck") pod "network-check-target-txjpp" (UID: "d4b74cf0-0931-4466-bc62-5b807071e600") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 16:58:36.694785 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:36.694713    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qp9lx"
Jul 01 16:58:36.694928 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:36.694827    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fp8j8"
Jul 01 16:58:36.694928 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:36.694841    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qp9lx" podUID="ab1a046b-4d82-418b-9c94-3700fa11dd87"
Jul 01 16:58:36.694928 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:36.694853    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-txjpp"
Jul 01 16:58:36.695045 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:36.694939    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fp8j8" podUID="899ca2bf-8e98-4def-a2f9-ace75d3170a0"
Jul 01 16:58:36.695045 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:36.694998    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-txjpp" podUID="d4b74cf0-0931-4466-bc62-5b807071e600"
Jul 01 16:58:38.691956 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:38.691744    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fp8j8"
Jul 01 16:58:38.692405 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:38.692056    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fp8j8" podUID="899ca2bf-8e98-4def-a2f9-ace75d3170a0"
Jul 01 16:58:38.692405 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:38.692282    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-txjpp"
Jul 01 16:58:38.692533 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:38.692405    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-txjpp" podUID="d4b74cf0-0931-4466-bc62-5b807071e600"
Jul 01 16:58:38.692533 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:38.692441    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qp9lx"
Jul 01 16:58:38.692533 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:38.692525    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qp9lx" podUID="ab1a046b-4d82-418b-9c94-3700fa11dd87"
Jul 01 16:58:39.018534 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:39.018505    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-180.ec2.internal" event="NodeReady"
Jul 01 16:58:39.018717 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:39.018644    2563 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 01 16:58:39.072558 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:39.072526    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-86ccb9dc9-trl2q"]
Jul 01 16:58:39.077815 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:39.077792    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-86ccb9dc9-trl2q"
Jul 01 16:58:39.080678 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:39.080654    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 01 16:58:39.080806 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:39.080780    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 01 16:58:39.080806 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:39.080796    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-8gxrw\""
Jul 01 16:58:39.080963 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:39.080916    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 01 16:58:39.087925 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:39.087894    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:58:39.088166 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:39.088151    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 01 16:58:39.103357 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:39.103325    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-86ccb9dc9-trl2q"]
Jul 01 16:58:39.103473 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:39.103405    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-wr6x6"]
Jul 01 16:58:39.107147 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:39.107126    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-dm2q7"]
Jul 01 16:58:39.107283 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:39.107264    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wr6x6"
Jul 01 16:58:39.110581 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:39.110560    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dm2q7"
Jul 01 16:58:39.112422 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:39.112403    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 01 16:58:39.112422 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:39.112417    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 01 16:58:39.112581 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:39.112481    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wm6rt\""
Jul 01 16:58:39.115704 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:39.115156    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 01 16:58:39.115704 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:39.115193    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-7mnn7\""
Jul 01 16:58:39.115704 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:39.115664    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 01 16:58:39.116045 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:39.115881    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 01 16:58:39.132210 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:39.132186    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dm2q7"]
Jul 01 16:58:39.142024 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:39.141994    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wr6x6"]
Jul 01 16:58:39.215656 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:39.215622    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/73d415ea-62f2-4da8-8610-93fbbff6cf60-installation-pull-secrets\") pod \"image-registry-86ccb9dc9-trl2q\" (UID: \"73d415ea-62f2-4da8-8610-93fbbff6cf60\") " pod="openshift-image-registry/image-registry-86ccb9dc9-trl2q"
Jul 01 16:58:39.215842 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:39.215673    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snxrw\" (UniqueName: \"kubernetes.io/projected/73d415ea-62f2-4da8-8610-93fbbff6cf60-kube-api-access-snxrw\") pod \"image-registry-86ccb9dc9-trl2q\" (UID: \"73d415ea-62f2-4da8-8610-93fbbff6cf60\") " pod="openshift-image-registry/image-registry-86ccb9dc9-trl2q"
Jul 01 16:58:39.215842 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:39.215697    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqwgq\" (UniqueName: \"kubernetes.io/projected/12b51164-c7a2-464b-8652-077ef495ff1f-kube-api-access-tqwgq\") pod \"ingress-canary-dm2q7\" (UID: \"12b51164-c7a2-464b-8652-077ef495ff1f\") " pod="openshift-ingress-canary/ingress-canary-dm2q7"
Jul 01 16:58:39.215842 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:39.215716    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73d415ea-62f2-4da8-8610-93fbbff6cf60-registry-tls\") pod \"image-registry-86ccb9dc9-trl2q\" (UID: \"73d415ea-62f2-4da8-8610-93fbbff6cf60\") " pod="openshift-image-registry/image-registry-86ccb9dc9-trl2q"
Jul 01 16:58:39.215842 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:39.215739    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/73d415ea-62f2-4da8-8610-93fbbff6cf60-registry-certificates\") pod \"image-registry-86ccb9dc9-trl2q\" (UID: \"73d415ea-62f2-4da8-8610-93fbbff6cf60\") " pod="openshift-image-registry/image-registry-86ccb9dc9-trl2q"
Jul 01 16:58:39.215842 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:39.215775    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6867bb01-cbf2-4ced-8a1e-a6fa15bfa8d7-config-volume\") pod \"dns-default-wr6x6\" (UID: \"6867bb01-cbf2-4ced-8a1e-a6fa15bfa8d7\") " pod="openshift-dns/dns-default-wr6x6"
Jul 01 16:58:39.216063 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:39.215857    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6867bb01-cbf2-4ced-8a1e-a6fa15bfa8d7-tmp-dir\") pod \"dns-default-wr6x6\" (UID: \"6867bb01-cbf2-4ced-8a1e-a6fa15bfa8d7\") " pod="openshift-dns/dns-default-wr6x6"
Jul 01 16:58:39.216063 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:39.215924    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/73d415ea-62f2-4da8-8610-93fbbff6cf60-image-registry-private-configuration\") pod \"image-registry-86ccb9dc9-trl2q\" (UID: \"73d415ea-62f2-4da8-8610-93fbbff6cf60\") " pod="openshift-image-registry/image-registry-86ccb9dc9-trl2q"
Jul 01 16:58:39.216063 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:39.215946    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t9hd\" (UniqueName: \"kubernetes.io/projected/6867bb01-cbf2-4ced-8a1e-a6fa15bfa8d7-kube-api-access-6t9hd\") pod \"dns-default-wr6x6\" (UID: \"6867bb01-cbf2-4ced-8a1e-a6fa15bfa8d7\") " pod="openshift-dns/dns-default-wr6x6"
Jul 01 16:58:39.216063 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:39.215972    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/73d415ea-62f2-4da8-8610-93fbbff6cf60-ca-trust-extracted\") pod \"image-registry-86ccb9dc9-trl2q\" (UID: \"73d415ea-62f2-4da8-8610-93fbbff6cf60\") " pod="openshift-image-registry/image-registry-86ccb9dc9-trl2q"
Jul 01 16:58:39.216063 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:39.215994    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/12b51164-c7a2-464b-8652-077ef495ff1f-cert\") pod \"ingress-canary-dm2q7\" (UID: \"12b51164-c7a2-464b-8652-077ef495ff1f\") " pod="openshift-ingress-canary/ingress-canary-dm2q7"
Jul 01 16:58:39.216063 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:39.216035    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6867bb01-cbf2-4ced-8a1e-a6fa15bfa8d7-metrics-tls\") pod \"dns-default-wr6x6\" (UID: \"6867bb01-cbf2-4ced-8a1e-a6fa15bfa8d7\") " pod="openshift-dns/dns-default-wr6x6"
Jul 01 16:58:39.216306 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:39.216114    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/73d415ea-62f2-4da8-8610-93fbbff6cf60-trusted-ca\") pod \"image-registry-86ccb9dc9-trl2q\" (UID: \"73d415ea-62f2-4da8-8610-93fbbff6cf60\") " pod="openshift-image-registry/image-registry-86ccb9dc9-trl2q"
Jul 01 16:58:39.216306 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:39.216139    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/73d415ea-62f2-4da8-8610-93fbbff6cf60-bound-sa-token\") pod \"image-registry-86ccb9dc9-trl2q\" (UID: \"73d415ea-62f2-4da8-8610-93fbbff6cf60\") " pod="openshift-image-registry/image-registry-86ccb9dc9-trl2q"
Jul 01 16:58:39.220278 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:39.220253    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:58:39.317274 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:39.317148    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/73d415ea-62f2-4da8-8610-93fbbff6cf60-trusted-ca\") pod \"image-registry-86ccb9dc9-trl2q\" (UID: \"73d415ea-62f2-4da8-8610-93fbbff6cf60\") " pod="openshift-image-registry/image-registry-86ccb9dc9-trl2q"
Jul 01 16:58:39.317274 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:39.317191    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/73d415ea-62f2-4da8-8610-93fbbff6cf60-bound-sa-token\") pod \"image-registry-86ccb9dc9-trl2q\" (UID: \"73d415ea-62f2-4da8-8610-93fbbff6cf60\") " pod="openshift-image-registry/image-registry-86ccb9dc9-trl2q"
Jul 01 16:58:39.317274 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:39.317227    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/73d415ea-62f2-4da8-8610-93fbbff6cf60-installation-pull-secrets\") pod \"image-registry-86ccb9dc9-trl2q\" (UID: \"73d415ea-62f2-4da8-8610-93fbbff6cf60\") " pod="openshift-image-registry/image-registry-86ccb9dc9-trl2q"
Jul 01 16:58:39.317274 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:39.317249    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snxrw\" (UniqueName: \"kubernetes.io/projected/73d415ea-62f2-4da8-8610-93fbbff6cf60-kube-api-access-snxrw\") pod \"image-registry-86ccb9dc9-trl2q\" (UID: \"73d415ea-62f2-4da8-8610-93fbbff6cf60\") " pod="openshift-image-registry/image-registry-86ccb9dc9-trl2q"
Jul 01 16:58:39.317274 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:39.317265    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqwgq\" (UniqueName: \"kubernetes.io/projected/12b51164-c7a2-464b-8652-077ef495ff1f-kube-api-access-tqwgq\") pod \"ingress-canary-dm2q7\" (UID: \"12b51164-c7a2-464b-8652-077ef495ff1f\") " pod="openshift-ingress-canary/ingress-canary-dm2q7"
Jul 01 16:58:39.317648 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:39.317288    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73d415ea-62f2-4da8-8610-93fbbff6cf60-registry-tls\") pod \"image-registry-86ccb9dc9-trl2q\" (UID: \"73d415ea-62f2-4da8-8610-93fbbff6cf60\") " pod="openshift-image-registry/image-registry-86ccb9dc9-trl2q"
Jul 01 16:58:39.317648 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:39.317310    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/73d415ea-62f2-4da8-8610-93fbbff6cf60-registry-certificates\") pod \"image-registry-86ccb9dc9-trl2q\" (UID: \"73d415ea-62f2-4da8-8610-93fbbff6cf60\") " pod="openshift-image-registry/image-registry-86ccb9dc9-trl2q"
Jul 01 16:58:39.317648 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:39.317326    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6867bb01-cbf2-4ced-8a1e-a6fa15bfa8d7-config-volume\") pod \"dns-default-wr6x6\" (UID: \"6867bb01-cbf2-4ced-8a1e-a6fa15bfa8d7\") " pod="openshift-dns/dns-default-wr6x6"
Jul 01 16:58:39.317648 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:39.317356    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6867bb01-cbf2-4ced-8a1e-a6fa15bfa8d7-tmp-dir\") pod \"dns-default-wr6x6\" (UID: \"6867bb01-cbf2-4ced-8a1e-a6fa15bfa8d7\") " pod="openshift-dns/dns-default-wr6x6"
Jul 01 16:58:39.317648 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:39.317383    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/73d415ea-62f2-4da8-8610-93fbbff6cf60-image-registry-private-configuration\") pod \"image-registry-86ccb9dc9-trl2q\" (UID: \"73d415ea-62f2-4da8-8610-93fbbff6cf60\") " pod="openshift-image-registry/image-registry-86ccb9dc9-trl2q"
Jul 01 16:58:39.317648 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:39.317404    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t9hd\" (UniqueName: \"kubernetes.io/projected/6867bb01-cbf2-4ced-8a1e-a6fa15bfa8d7-kube-api-access-6t9hd\") pod \"dns-default-wr6x6\" (UID: \"6867bb01-cbf2-4ced-8a1e-a6fa15bfa8d7\") " pod="openshift-dns/dns-default-wr6x6"
Jul 01 16:58:39.317648 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:39.317440    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/73d415ea-62f2-4da8-8610-93fbbff6cf60-ca-trust-extracted\") pod \"image-registry-86ccb9dc9-trl2q\" (UID: \"73d415ea-62f2-4da8-8610-93fbbff6cf60\") " pod="openshift-image-registry/image-registry-86ccb9dc9-trl2q"
Jul 01 16:58:39.317648 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:39.317463    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/12b51164-c7a2-464b-8652-077ef495ff1f-cert\") pod \"ingress-canary-dm2q7\" (UID: \"12b51164-c7a2-464b-8652-077ef495ff1f\") " pod="openshift-ingress-canary/ingress-canary-dm2q7"
Jul 01 16:58:39.317648 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:39.317495    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6867bb01-cbf2-4ced-8a1e-a6fa15bfa8d7-metrics-tls\") pod \"dns-default-wr6x6\" (UID: \"6867bb01-cbf2-4ced-8a1e-a6fa15bfa8d7\") " pod="openshift-dns/dns-default-wr6x6"
Jul 01 16:58:39.317648 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:39.317605    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 16:58:39.318109 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:39.317673    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6867bb01-cbf2-4ced-8a1e-a6fa15bfa8d7-metrics-tls podName:6867bb01-cbf2-4ced-8a1e-a6fa15bfa8d7 nodeName:}" failed. No retries permitted until 2026-07-01 16:58:39.817651924 +0000 UTC m=+37.614790310 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6867bb01-cbf2-4ced-8a1e-a6fa15bfa8d7-metrics-tls") pod "dns-default-wr6x6" (UID: "6867bb01-cbf2-4ced-8a1e-a6fa15bfa8d7") : secret "dns-default-metrics-tls" not found
Jul 01 16:58:39.318109 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:39.318082    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6867bb01-cbf2-4ced-8a1e-a6fa15bfa8d7-tmp-dir\") pod \"dns-default-wr6x6\" (UID: \"6867bb01-cbf2-4ced-8a1e-a6fa15bfa8d7\") " pod="openshift-dns/dns-default-wr6x6"
Jul 01 16:58:39.318217 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:39.318133    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/73d415ea-62f2-4da8-8610-93fbbff6cf60-trusted-ca\") pod \"image-registry-86ccb9dc9-trl2q\" (UID: \"73d415ea-62f2-4da8-8610-93fbbff6cf60\") " pod="openshift-image-registry/image-registry-86ccb9dc9-trl2q"
Jul 01 16:58:39.318217 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:39.318142    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6867bb01-cbf2-4ced-8a1e-a6fa15bfa8d7-config-volume\") pod \"dns-default-wr6x6\" (UID: \"6867bb01-cbf2-4ced-8a1e-a6fa15bfa8d7\") " pod="openshift-dns/dns-default-wr6x6"
Jul 01 16:58:39.318488 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:39.318462    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 16:58:39.318488 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:39.318486    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86ccb9dc9-trl2q: secret "image-registry-tls" not found
Jul 01 16:58:39.318672 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:39.318538    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73d415ea-62f2-4da8-8610-93fbbff6cf60-registry-tls podName:73d415ea-62f2-4da8-8610-93fbbff6cf60 nodeName:}" failed. No retries permitted until 2026-07-01 16:58:39.81851982 +0000 UTC m=+37.615658201 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/73d415ea-62f2-4da8-8610-93fbbff6cf60-registry-tls") pod "image-registry-86ccb9dc9-trl2q" (UID: "73d415ea-62f2-4da8-8610-93fbbff6cf60") : secret "image-registry-tls" not found
Jul 01 16:58:39.318672 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:39.318621    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/73d415ea-62f2-4da8-8610-93fbbff6cf60-ca-trust-extracted\") pod \"image-registry-86ccb9dc9-trl2q\" (UID: \"73d415ea-62f2-4da8-8610-93fbbff6cf60\") " pod="openshift-image-registry/image-registry-86ccb9dc9-trl2q"
Jul 01 16:58:39.318876 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:39.318821    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/73d415ea-62f2-4da8-8610-93fbbff6cf60-registry-certificates\") pod \"image-registry-86ccb9dc9-trl2q\" (UID: \"73d415ea-62f2-4da8-8610-93fbbff6cf60\") " pod="openshift-image-registry/image-registry-86ccb9dc9-trl2q"
Jul 01 16:58:39.322917 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:39.322895    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/12b51164-c7a2-464b-8652-077ef495ff1f-cert\") pod \"ingress-canary-dm2q7\" (UID: \"12b51164-c7a2-464b-8652-077ef495ff1f\") " pod="openshift-ingress-canary/ingress-canary-dm2q7"
Jul 01 16:58:39.323047 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:39.322898    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/73d415ea-62f2-4da8-8610-93fbbff6cf60-installation-pull-secrets\") pod \"image-registry-86ccb9dc9-trl2q\" (UID: \"73d415ea-62f2-4da8-8610-93fbbff6cf60\") " pod="openshift-image-registry/image-registry-86ccb9dc9-trl2q"
Jul 01 16:58:39.323245 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:39.323225    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/73d415ea-62f2-4da8-8610-93fbbff6cf60-image-registry-private-configuration\") pod \"image-registry-86ccb9dc9-trl2q\" (UID: \"73d415ea-62f2-4da8-8610-93fbbff6cf60\") " pod="openshift-image-registry/image-registry-86ccb9dc9-trl2q"
Jul 01 16:58:39.328193 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:39.328157    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t9hd\" (UniqueName: \"kubernetes.io/projected/6867bb01-cbf2-4ced-8a1e-a6fa15bfa8d7-kube-api-access-6t9hd\") pod \"dns-default-wr6x6\" (UID: \"6867bb01-cbf2-4ced-8a1e-a6fa15bfa8d7\") " pod="openshift-dns/dns-default-wr6x6"
Jul 01 16:58:39.328545 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:39.328522    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/73d415ea-62f2-4da8-8610-93fbbff6cf60-bound-sa-token\") pod \"image-registry-86ccb9dc9-trl2q\" (UID: \"73d415ea-62f2-4da8-8610-93fbbff6cf60\") " pod="openshift-image-registry/image-registry-86ccb9dc9-trl2q"
Jul 01 16:58:39.330066 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:39.330047    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqwgq\" (UniqueName: \"kubernetes.io/projected/12b51164-c7a2-464b-8652-077ef495ff1f-kube-api-access-tqwgq\") pod \"ingress-canary-dm2q7\" (UID: \"12b51164-c7a2-464b-8652-077ef495ff1f\") " pod="openshift-ingress-canary/ingress-canary-dm2q7"
Jul 01 16:58:39.331007 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:39.330984    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snxrw\" (UniqueName: \"kubernetes.io/projected/73d415ea-62f2-4da8-8610-93fbbff6cf60-kube-api-access-snxrw\") pod \"image-registry-86ccb9dc9-trl2q\" (UID: \"73d415ea-62f2-4da8-8610-93fbbff6cf60\") " pod="openshift-image-registry/image-registry-86ccb9dc9-trl2q"
Jul 01 16:58:39.425017 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:39.424986    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dm2q7"
Jul 01 16:58:39.446458 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:39.446424    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:58:39.547291 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:39.547254    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:58:39.821899 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:39.821864    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73d415ea-62f2-4da8-8610-93fbbff6cf60-registry-tls\") pod \"image-registry-86ccb9dc9-trl2q\" (UID: \"73d415ea-62f2-4da8-8610-93fbbff6cf60\") " pod="openshift-image-registry/image-registry-86ccb9dc9-trl2q"
Jul 01 16:58:39.822325 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:39.821927    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6867bb01-cbf2-4ced-8a1e-a6fa15bfa8d7-metrics-tls\") pod \"dns-default-wr6x6\" (UID: \"6867bb01-cbf2-4ced-8a1e-a6fa15bfa8d7\") " pod="openshift-dns/dns-default-wr6x6"
Jul 01 16:58:39.822325 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:39.822037    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 16:58:39.822325 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:39.822042    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 16:58:39.822325 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:39.822067    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86ccb9dc9-trl2q: secret "image-registry-tls" not found
Jul 01 16:58:39.822325 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:39.822123    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6867bb01-cbf2-4ced-8a1e-a6fa15bfa8d7-metrics-tls podName:6867bb01-cbf2-4ced-8a1e-a6fa15bfa8d7 nodeName:}" failed. No retries permitted until 2026-07-01 16:58:40.822103188 +0000 UTC m=+38.619241590 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6867bb01-cbf2-4ced-8a1e-a6fa15bfa8d7-metrics-tls") pod "dns-default-wr6x6" (UID: "6867bb01-cbf2-4ced-8a1e-a6fa15bfa8d7") : secret "dns-default-metrics-tls" not found
Jul 01 16:58:39.822325 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:39.822143    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73d415ea-62f2-4da8-8610-93fbbff6cf60-registry-tls podName:73d415ea-62f2-4da8-8610-93fbbff6cf60 nodeName:}" failed. No retries permitted until 2026-07-01 16:58:40.822133405 +0000 UTC m=+38.619271786 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/73d415ea-62f2-4da8-8610-93fbbff6cf60-registry-tls") pod "image-registry-86ccb9dc9-trl2q" (UID: "73d415ea-62f2-4da8-8610-93fbbff6cf60") : secret "image-registry-tls" not found
Jul 01 16:58:40.094720 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:40.094688    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dm2q7"]
Jul 01 16:58:40.100214 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:58:40.100182    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12b51164_c7a2_464b_8652_077ef495ff1f.slice/crio-1604ce76317dfcbe32e2cb71ff8adb10643686af866d63b3be4780f8cd713e6b WatchSource:0}: Error finding container 1604ce76317dfcbe32e2cb71ff8adb10643686af866d63b3be4780f8cd713e6b: Status 404 returned error can't find the container with id 1604ce76317dfcbe32e2cb71ff8adb10643686af866d63b3be4780f8cd713e6b
Jul 01 16:58:40.691227 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:40.691190    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qp9lx"
Jul 01 16:58:40.691227 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:40.691218    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fp8j8"
Jul 01 16:58:40.691562 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:40.691536    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-txjpp"
Jul 01 16:58:40.694536 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:40.694512    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 01 16:58:40.695776 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:40.695755    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 01 16:58:40.695883 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:40.695778    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6c472\""
Jul 01 16:58:40.695883 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:40.695849    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 01 16:58:40.695993 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:40.695931    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 01 16:58:40.696043 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:40.696034    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-6lq7p\""
Jul 01 16:58:40.830485 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:40.830458    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73d415ea-62f2-4da8-8610-93fbbff6cf60-registry-tls\") pod \"image-registry-86ccb9dc9-trl2q\" (UID: \"73d415ea-62f2-4da8-8610-93fbbff6cf60\") " pod="openshift-image-registry/image-registry-86ccb9dc9-trl2q"
Jul 01 16:58:40.830896 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:40.830508    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6867bb01-cbf2-4ced-8a1e-a6fa15bfa8d7-metrics-tls\") pod \"dns-default-wr6x6\" (UID: \"6867bb01-cbf2-4ced-8a1e-a6fa15bfa8d7\") " pod="openshift-dns/dns-default-wr6x6"
Jul 01 16:58:40.830896 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:40.830623    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 16:58:40.830896 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:40.830681    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6867bb01-cbf2-4ced-8a1e-a6fa15bfa8d7-metrics-tls podName:6867bb01-cbf2-4ced-8a1e-a6fa15bfa8d7 nodeName:}" failed. No retries permitted until 2026-07-01 16:58:42.830667881 +0000 UTC m=+40.627806260 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6867bb01-cbf2-4ced-8a1e-a6fa15bfa8d7-metrics-tls") pod "dns-default-wr6x6" (UID: "6867bb01-cbf2-4ced-8a1e-a6fa15bfa8d7") : secret "dns-default-metrics-tls" not found
Jul 01 16:58:40.831032 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:40.830909    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 16:58:40.831032 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:40.830924    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86ccb9dc9-trl2q: secret "image-registry-tls" not found
Jul 01 16:58:40.831032 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:40.830971    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73d415ea-62f2-4da8-8610-93fbbff6cf60-registry-tls podName:73d415ea-62f2-4da8-8610-93fbbff6cf60 nodeName:}" failed. No retries permitted until 2026-07-01 16:58:42.830960021 +0000 UTC m=+40.628098400 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/73d415ea-62f2-4da8-8610-93fbbff6cf60-registry-tls") pod "image-registry-86ccb9dc9-trl2q" (UID: "73d415ea-62f2-4da8-8610-93fbbff6cf60") : secret "image-registry-tls" not found
Jul 01 16:58:40.923922 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:40.923744    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dm2q7" event={"ID":"12b51164-c7a2-464b-8652-077ef495ff1f","Type":"ContainerStarted","Data":"1604ce76317dfcbe32e2cb71ff8adb10643686af866d63b3be4780f8cd713e6b"}
Jul 01 16:58:40.927636 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:40.927609    2563 generic.go:356] "Generic (PLEG): container finished" podID="0ab87fa2-edd2-4c8b-a666-1efa2c3bae8e" containerID="e001246f2908fb9a86400c8da44bf5b6216ce7752545366795fca942ae554a07" exitCode=0
Jul 01 16:58:40.927770 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:40.927645    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gh58z" event={"ID":"0ab87fa2-edd2-4c8b-a666-1efa2c3bae8e","Type":"ContainerDied","Data":"e001246f2908fb9a86400c8da44bf5b6216ce7752545366795fca942ae554a07"}
Jul 01 16:58:41.933031 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:41.932984    2563 generic.go:356] "Generic (PLEG): container finished" podID="0ab87fa2-edd2-4c8b-a666-1efa2c3bae8e" containerID="05f2855b10b4e9a786964302d2f75d29df99370bcf55a57cb79b98a3428e8968" exitCode=0
Jul 01 16:58:41.933392 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:41.933037    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gh58z" event={"ID":"0ab87fa2-edd2-4c8b-a666-1efa2c3bae8e","Type":"ContainerDied","Data":"05f2855b10b4e9a786964302d2f75d29df99370bcf55a57cb79b98a3428e8968"}
Jul 01 16:58:42.573110 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:42.573075    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:58:42.720569 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:42.720536    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:58:42.844440 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:42.844405    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73d415ea-62f2-4da8-8610-93fbbff6cf60-registry-tls\") pod \"image-registry-86ccb9dc9-trl2q\" (UID: \"73d415ea-62f2-4da8-8610-93fbbff6cf60\") " pod="openshift-image-registry/image-registry-86ccb9dc9-trl2q"
Jul 01 16:58:42.844605 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:42.844457    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6867bb01-cbf2-4ced-8a1e-a6fa15bfa8d7-metrics-tls\") pod \"dns-default-wr6x6\" (UID: \"6867bb01-cbf2-4ced-8a1e-a6fa15bfa8d7\") " pod="openshift-dns/dns-default-wr6x6"
Jul 01 16:58:42.844605 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:42.844569    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 16:58:42.844605 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:42.844593    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86ccb9dc9-trl2q: secret "image-registry-tls" not found
Jul 01 16:58:42.844710 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:42.844646    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73d415ea-62f2-4da8-8610-93fbbff6cf60-registry-tls podName:73d415ea-62f2-4da8-8610-93fbbff6cf60 nodeName:}" failed. No retries permitted until 2026-07-01 16:58:46.844632167 +0000 UTC m=+44.641770545 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/73d415ea-62f2-4da8-8610-93fbbff6cf60-registry-tls") pod "image-registry-86ccb9dc9-trl2q" (UID: "73d415ea-62f2-4da8-8610-93fbbff6cf60") : secret "image-registry-tls" not found
Jul 01 16:58:42.844710 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:42.844573    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 16:58:42.844710 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:42.844703    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6867bb01-cbf2-4ced-8a1e-a6fa15bfa8d7-metrics-tls podName:6867bb01-cbf2-4ced-8a1e-a6fa15bfa8d7 nodeName:}" failed. No retries permitted until 2026-07-01 16:58:46.844691852 +0000 UTC m=+44.641830230 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6867bb01-cbf2-4ced-8a1e-a6fa15bfa8d7-metrics-tls") pod "dns-default-wr6x6" (UID: "6867bb01-cbf2-4ced-8a1e-a6fa15bfa8d7") : secret "dns-default-metrics-tls" not found
Jul 01 16:58:42.937157 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:42.937117    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dm2q7" event={"ID":"12b51164-c7a2-464b-8652-077ef495ff1f","Type":"ContainerStarted","Data":"9f8f8f6edd03a9d0879dd96ca7d5bf938ab1cb476203588e80d4ba7eecf026ac"}
Jul 01 16:58:42.940328 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:42.940304    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gh58z" event={"ID":"0ab87fa2-edd2-4c8b-a666-1efa2c3bae8e","Type":"ContainerStarted","Data":"ae5a7a43c312de9d2219138424f68fafff0a05207bd861649fde6043e84ae09b"}
Jul 01 16:58:42.958814 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:42.958769    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-dm2q7" podStartSLOduration=1.428472037 podStartE2EDuration="3.958751598s" totalImagesPullingTime="2.530279561s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 16:58:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 16:58:42.958231501 +0000 UTC m=+40.755369902" watchObservedRunningTime="2026-07-01 16:58:42.958751598 +0000 UTC m=+40.755890000"
Jul 01 16:58:42.982840 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:42.982742    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-gh58z" podStartSLOduration=9.646114419 podStartE2EDuration="39.982727863s" totalImagesPullingTime="30.336613444s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 16:58:03 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 16:58:42.98268099 +0000 UTC m=+40.779819392" watchObservedRunningTime="2026-07-01 16:58:42.982727863 +0000 UTC m=+40.779866263"
Jul 01 16:58:43.118178 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:43.118140    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:58:43.219728 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:43.219692    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:58:46.874518 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:46.874483    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73d415ea-62f2-4da8-8610-93fbbff6cf60-registry-tls\") pod \"image-registry-86ccb9dc9-trl2q\" (UID: \"73d415ea-62f2-4da8-8610-93fbbff6cf60\") " pod="openshift-image-registry/image-registry-86ccb9dc9-trl2q"
Jul 01 16:58:46.874931 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:46.874547    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6867bb01-cbf2-4ced-8a1e-a6fa15bfa8d7-metrics-tls\") pod \"dns-default-wr6x6\" (UID: \"6867bb01-cbf2-4ced-8a1e-a6fa15bfa8d7\") " pod="openshift-dns/dns-default-wr6x6"
Jul 01 16:58:46.874931 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:46.874627    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 16:58:46.874931 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:46.874650    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86ccb9dc9-trl2q: secret "image-registry-tls" not found
Jul 01 16:58:46.874931 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:46.874687    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 16:58:46.874931 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:46.874705    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73d415ea-62f2-4da8-8610-93fbbff6cf60-registry-tls podName:73d415ea-62f2-4da8-8610-93fbbff6cf60 nodeName:}" failed. No retries permitted until 2026-07-01 16:58:54.874690883 +0000 UTC m=+52.671829262 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/73d415ea-62f2-4da8-8610-93fbbff6cf60-registry-tls") pod "image-registry-86ccb9dc9-trl2q" (UID: "73d415ea-62f2-4da8-8610-93fbbff6cf60") : secret "image-registry-tls" not found
Jul 01 16:58:46.874931 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:46.874737    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6867bb01-cbf2-4ced-8a1e-a6fa15bfa8d7-metrics-tls podName:6867bb01-cbf2-4ced-8a1e-a6fa15bfa8d7 nodeName:}" failed. No retries permitted until 2026-07-01 16:58:54.874723201 +0000 UTC m=+52.671861584 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6867bb01-cbf2-4ced-8a1e-a6fa15bfa8d7-metrics-tls") pod "dns-default-wr6x6" (UID: "6867bb01-cbf2-4ced-8a1e-a6fa15bfa8d7") : secret "dns-default-metrics-tls" not found
Jul 01 16:58:54.749507 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:54.749468    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:58:54.847051 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:54.847011    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:58:54.923465 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:54.923420    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:58:54.930332 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:54.930307    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73d415ea-62f2-4da8-8610-93fbbff6cf60-registry-tls\") pod \"image-registry-86ccb9dc9-trl2q\" (UID: \"73d415ea-62f2-4da8-8610-93fbbff6cf60\") " pod="openshift-image-registry/image-registry-86ccb9dc9-trl2q"
Jul 01 16:58:54.930418 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:54.930367    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6867bb01-cbf2-4ced-8a1e-a6fa15bfa8d7-metrics-tls\") pod \"dns-default-wr6x6\" (UID: \"6867bb01-cbf2-4ced-8a1e-a6fa15bfa8d7\") " pod="openshift-dns/dns-default-wr6x6"
Jul 01 16:58:54.930474 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:54.930460    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 16:58:54.930512 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:54.930478    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86ccb9dc9-trl2q: secret "image-registry-tls" not found
Jul 01 16:58:54.930512 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:54.930459    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 16:58:54.930571 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:54.930529    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73d415ea-62f2-4da8-8610-93fbbff6cf60-registry-tls podName:73d415ea-62f2-4da8-8610-93fbbff6cf60 nodeName:}" failed. No retries permitted until 2026-07-01 16:59:10.930514331 +0000 UTC m=+68.727652710 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/73d415ea-62f2-4da8-8610-93fbbff6cf60-registry-tls") pod "image-registry-86ccb9dc9-trl2q" (UID: "73d415ea-62f2-4da8-8610-93fbbff6cf60") : secret "image-registry-tls" not found
Jul 01 16:58:54.930571 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:58:54.930542    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6867bb01-cbf2-4ced-8a1e-a6fa15bfa8d7-metrics-tls podName:6867bb01-cbf2-4ced-8a1e-a6fa15bfa8d7 nodeName:}" failed. No retries permitted until 2026-07-01 16:59:10.930536247 +0000 UTC m=+68.727674626 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6867bb01-cbf2-4ced-8a1e-a6fa15bfa8d7-metrics-tls") pod "dns-default-wr6x6" (UID: "6867bb01-cbf2-4ced-8a1e-a6fa15bfa8d7") : secret "dns-default-metrics-tls" not found
Jul 01 16:58:55.683702 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:55.683664    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:58:55.781600 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:55.781545    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:58:56.622836 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:58:56.622779    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:59:00.252434 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:00.252397    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:59:00.353162 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:00.353127    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:59:01.355615 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:01.355581    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:59:01.456957 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:01.456923    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:59:01.527004 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:01.526958    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:59:02.213155 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:02.213109    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:59:02.311397 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:02.311362    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:59:02.412007 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:02.411974    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:59:03.915359 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:03.915322    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j6kft"
Jul 01 16:59:04.101947 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:04.101900    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:59:04.211868 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:04.211783    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:59:06.785279 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:06.785238    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:59:06.883225 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:06.883189    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:59:08.433679 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:08.433643    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/899ca2bf-8e98-4def-a2f9-ace75d3170a0-metrics-certs\") pod \"network-metrics-daemon-fp8j8\" (UID: \"899ca2bf-8e98-4def-a2f9-ace75d3170a0\") " pod="openshift-multus/network-metrics-daemon-fp8j8"
Jul 01 16:59:08.436822 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:08.436805    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 01 16:59:08.444221 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:59:08.444203    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 01 16:59:08.444281 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:59:08.444258    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/899ca2bf-8e98-4def-a2f9-ace75d3170a0-metrics-certs podName:899ca2bf-8e98-4def-a2f9-ace75d3170a0 nodeName:}" failed. No retries permitted until 2026-07-01 17:00:12.444243219 +0000 UTC m=+130.241381598 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/899ca2bf-8e98-4def-a2f9-ace75d3170a0-metrics-certs") pod "network-metrics-daemon-fp8j8" (UID: "899ca2bf-8e98-4def-a2f9-ace75d3170a0") : secret "metrics-daemon-secret" not found
Jul 01 16:59:08.534136 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:08.534112    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkwck\" (UniqueName: \"kubernetes.io/projected/d4b74cf0-0931-4466-bc62-5b807071e600-kube-api-access-bkwck\") pod \"network-check-target-txjpp\" (UID: \"d4b74cf0-0931-4466-bc62-5b807071e600\") " pod="openshift-network-diagnostics/network-check-target-txjpp"
Jul 01 16:59:08.534222 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:08.534190    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab1a046b-4d82-418b-9c94-3700fa11dd87-original-pull-secret\") pod \"global-pull-secret-syncer-qp9lx\" (UID: \"ab1a046b-4d82-418b-9c94-3700fa11dd87\") " pod="kube-system/global-pull-secret-syncer-qp9lx"
Jul 01 16:59:08.537695 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:08.537669    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 01 16:59:08.537787 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:08.537669    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 01 16:59:08.547153 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:08.547133    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 01 16:59:08.548007 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:08.547989    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab1a046b-4d82-418b-9c94-3700fa11dd87-original-pull-secret\") pod \"global-pull-secret-syncer-qp9lx\" (UID: \"ab1a046b-4d82-418b-9c94-3700fa11dd87\") " pod="kube-system/global-pull-secret-syncer-qp9lx"
Jul 01 16:59:08.557710 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:08.557685    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkwck\" (UniqueName: \"kubernetes.io/projected/d4b74cf0-0931-4466-bc62-5b807071e600-kube-api-access-bkwck\") pod \"network-check-target-txjpp\" (UID: \"d4b74cf0-0931-4466-bc62-5b807071e600\") " pod="openshift-network-diagnostics/network-check-target-txjpp"
Jul 01 16:59:08.609082 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:08.609050    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qp9lx"
Jul 01 16:59:08.618691 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:08.618666    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-6lq7p\""
Jul 01 16:59:08.625605 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:08.625578    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-txjpp"
Jul 01 16:59:08.760132 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:08.760103    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qp9lx"]
Jul 01 16:59:08.764694 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:59:08.764663    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab1a046b_4d82_418b_9c94_3700fa11dd87.slice/crio-8e13eeb980c3b7b2de2c240cf0494af76b48fed35dee421045bfbc4e97015bad WatchSource:0}: Error finding container 8e13eeb980c3b7b2de2c240cf0494af76b48fed35dee421045bfbc4e97015bad: Status 404 returned error can't find the container with id 8e13eeb980c3b7b2de2c240cf0494af76b48fed35dee421045bfbc4e97015bad
Jul 01 16:59:08.787875 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:59:08.787851    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4b74cf0_0931_4466_bc62_5b807071e600.slice/crio-0b2389d3d644e63ce1d238a818a13a1dcae676942ad3fd28309e9615de80df01 WatchSource:0}: Error finding container 0b2389d3d644e63ce1d238a818a13a1dcae676942ad3fd28309e9615de80df01: Status 404 returned error can't find the container with id 0b2389d3d644e63ce1d238a818a13a1dcae676942ad3fd28309e9615de80df01
Jul 01 16:59:08.787997 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:08.787978    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-txjpp"]
Jul 01 16:59:08.999953 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:08.999923    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-txjpp" event={"ID":"d4b74cf0-0931-4466-bc62-5b807071e600","Type":"ContainerStarted","Data":"0b2389d3d644e63ce1d238a818a13a1dcae676942ad3fd28309e9615de80df01"}
Jul 01 16:59:09.001007 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:09.000982    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qp9lx" event={"ID":"ab1a046b-4d82-418b-9c94-3700fa11dd87","Type":"ContainerStarted","Data":"8e13eeb980c3b7b2de2c240cf0494af76b48fed35dee421045bfbc4e97015bad"}
Jul 01 16:59:09.830923 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:09.830888    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:59:10.006864 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:10.006779    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:59:10.954635 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:10.954601    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6867bb01-cbf2-4ced-8a1e-a6fa15bfa8d7-metrics-tls\") pod \"dns-default-wr6x6\" (UID: \"6867bb01-cbf2-4ced-8a1e-a6fa15bfa8d7\") " pod="openshift-dns/dns-default-wr6x6"
Jul 01 16:59:10.955096 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:10.954693    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73d415ea-62f2-4da8-8610-93fbbff6cf60-registry-tls\") pod \"image-registry-86ccb9dc9-trl2q\" (UID: \"73d415ea-62f2-4da8-8610-93fbbff6cf60\") " pod="openshift-image-registry/image-registry-86ccb9dc9-trl2q"
Jul 01 16:59:10.955096 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:59:10.954772    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 16:59:10.955096 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:59:10.954826    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 16:59:10.955096 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:59:10.954839    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86ccb9dc9-trl2q: secret "image-registry-tls" not found
Jul 01 16:59:10.955096 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:59:10.954845    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6867bb01-cbf2-4ced-8a1e-a6fa15bfa8d7-metrics-tls podName:6867bb01-cbf2-4ced-8a1e-a6fa15bfa8d7 nodeName:}" failed. No retries permitted until 2026-07-01 16:59:42.954823181 +0000 UTC m=+100.751961566 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6867bb01-cbf2-4ced-8a1e-a6fa15bfa8d7-metrics-tls") pod "dns-default-wr6x6" (UID: "6867bb01-cbf2-4ced-8a1e-a6fa15bfa8d7") : secret "dns-default-metrics-tls" not found
Jul 01 16:59:10.955096 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:59:10.954876    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73d415ea-62f2-4da8-8610-93fbbff6cf60-registry-tls podName:73d415ea-62f2-4da8-8610-93fbbff6cf60 nodeName:}" failed. No retries permitted until 2026-07-01 16:59:42.954864504 +0000 UTC m=+100.752002883 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/73d415ea-62f2-4da8-8610-93fbbff6cf60-registry-tls") pod "image-registry-86ccb9dc9-trl2q" (UID: "73d415ea-62f2-4da8-8610-93fbbff6cf60") : secret "image-registry-tls" not found
Jul 01 16:59:11.867502 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:11.867459    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:59:12.077018 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:12.076979    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:59:13.360449 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:13.360359    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:59:13.648967 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:13.648880    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:59:14.015522 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:14.015484    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-txjpp" event={"ID":"d4b74cf0-0931-4466-bc62-5b807071e600","Type":"ContainerStarted","Data":"75c89da18f567a880929c830656c9daafc20aaf1f6953444a799e87414b5638f"}
Jul 01 16:59:14.015754 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:14.015607    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-txjpp"
Jul 01 16:59:14.016968 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:14.016943    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qp9lx" event={"ID":"ab1a046b-4d82-418b-9c94-3700fa11dd87","Type":"ContainerStarted","Data":"63b5f505c9b2613443689b07ec9957beee7fbe3bf44e12c5c1ace8bc9b545bbc"}
Jul 01 16:59:14.052815 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:14.052765    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-txjpp" podStartSLOduration=66.035476569 podStartE2EDuration="1m11.052746329s" totalImagesPullingTime="5.01726976s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 16:58:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 16:59:14.05267475 +0000 UTC m=+71.849813150" watchObservedRunningTime="2026-07-01 16:59:14.052746329 +0000 UTC m=+71.849884729"
Jul 01 16:59:17.830082 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:17.830041    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:59:19.327629 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:19.327592    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:59:21.358830 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:21.358784    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:59:32.730242 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:32.730202    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:59:32.830135 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:32.830095    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:59:36.874919 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:36.874884    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:59:36.949280 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:36.949247    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:59:39.386126 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:39.386087    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:59:39.485395 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:39.485336    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:59:42.567691 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:42.567637    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-qp9lx" podStartSLOduration=94.522812744 podStartE2EDuration="1m39.56762153s" totalImagesPullingTime="5.044808786s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 16:58:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 16:59:14.100498073 +0000 UTC m=+71.897636476" watchObservedRunningTime="2026-07-01 16:59:42.56762153 +0000 UTC m=+100.364759909"
Jul 01 16:59:42.568463 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:42.568436    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-95stp"]
Jul 01 16:59:42.572652 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:42.572632    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-95stp"
Jul 01 16:59:42.576781 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:42.576759    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 01 16:59:42.576885 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:42.576867    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-97p9s\""
Jul 01 16:59:42.576936 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:42.576898    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 01 16:59:42.577981 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:42.577959    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 01 16:59:42.578065 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:42.578003    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 01 16:59:42.610948 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:42.610920    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-95stp"]
Jul 01 16:59:42.675467 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:42.675438    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c0a17941-c9cd-4865-93e1-0a07014a2857-serving-cert\") pod \"service-ca-operator-5d7b7dc95c-95stp\" (UID: \"c0a17941-c9cd-4865-93e1-0a07014a2857\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-95stp"
Jul 01 16:59:42.675607 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:42.675490    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2smd\" (UniqueName: \"kubernetes.io/projected/c0a17941-c9cd-4865-93e1-0a07014a2857-kube-api-access-k2smd\") pod \"service-ca-operator-5d7b7dc95c-95stp\" (UID: \"c0a17941-c9cd-4865-93e1-0a07014a2857\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-95stp"
Jul 01 16:59:42.675607 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:42.675516    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c0a17941-c9cd-4865-93e1-0a07014a2857-config\") pod \"service-ca-operator-5d7b7dc95c-95stp\" (UID: \"c0a17941-c9cd-4865-93e1-0a07014a2857\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-95stp"
Jul 01 16:59:42.676515 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:42.676497    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-hxfv2"]
Jul 01 16:59:42.679243 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:42.679225    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-hxfv2"
Jul 01 16:59:42.684645 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:42.684624    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 01 16:59:42.685047 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:42.684963    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-qpkfs\""
Jul 01 16:59:42.689177 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:42.689160    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 01 16:59:42.726659 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:42.726635    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-hxfv2"]
Jul 01 16:59:42.775899 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:42.775862    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c0a17941-c9cd-4865-93e1-0a07014a2857-serving-cert\") pod \"service-ca-operator-5d7b7dc95c-95stp\" (UID: \"c0a17941-c9cd-4865-93e1-0a07014a2857\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-95stp"
Jul 01 16:59:42.776043 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:42.775921    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2smd\" (UniqueName: \"kubernetes.io/projected/c0a17941-c9cd-4865-93e1-0a07014a2857-kube-api-access-k2smd\") pod \"service-ca-operator-5d7b7dc95c-95stp\" (UID: \"c0a17941-c9cd-4865-93e1-0a07014a2857\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-95stp"
Jul 01 16:59:42.776043 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:42.775945    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss4rm\" (UniqueName: \"kubernetes.io/projected/e9590c1d-88ae-4902-b0ed-2ce2cf13435b-kube-api-access-ss4rm\") pod \"volume-data-source-validator-69d9f679db-hxfv2\" (UID: \"e9590c1d-88ae-4902-b0ed-2ce2cf13435b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-hxfv2"
Jul 01 16:59:42.776043 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:42.775976    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c0a17941-c9cd-4865-93e1-0a07014a2857-config\") pod \"service-ca-operator-5d7b7dc95c-95stp\" (UID: \"c0a17941-c9cd-4865-93e1-0a07014a2857\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-95stp"
Jul 01 16:59:42.777255 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:42.777233    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c0a17941-c9cd-4865-93e1-0a07014a2857-config\") pod \"service-ca-operator-5d7b7dc95c-95stp\" (UID: \"c0a17941-c9cd-4865-93e1-0a07014a2857\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-95stp"
Jul 01 16:59:42.778257 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:42.778238    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c0a17941-c9cd-4865-93e1-0a07014a2857-serving-cert\") pod \"service-ca-operator-5d7b7dc95c-95stp\" (UID: \"c0a17941-c9cd-4865-93e1-0a07014a2857\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-95stp"
Jul 01 16:59:42.799092 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:42.799070    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2smd\" (UniqueName: \"kubernetes.io/projected/c0a17941-c9cd-4865-93e1-0a07014a2857-kube-api-access-k2smd\") pod \"service-ca-operator-5d7b7dc95c-95stp\" (UID: \"c0a17941-c9cd-4865-93e1-0a07014a2857\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-95stp"
Jul 01 16:59:42.876462 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:42.876365    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss4rm\" (UniqueName: \"kubernetes.io/projected/e9590c1d-88ae-4902-b0ed-2ce2cf13435b-kube-api-access-ss4rm\") pod \"volume-data-source-validator-69d9f679db-hxfv2\" (UID: \"e9590c1d-88ae-4902-b0ed-2ce2cf13435b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-hxfv2"
Jul 01 16:59:42.880684 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:42.880667    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-95stp"
Jul 01 16:59:42.888877 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:42.888852    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss4rm\" (UniqueName: \"kubernetes.io/projected/e9590c1d-88ae-4902-b0ed-2ce2cf13435b-kube-api-access-ss4rm\") pod \"volume-data-source-validator-69d9f679db-hxfv2\" (UID: \"e9590c1d-88ae-4902-b0ed-2ce2cf13435b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-hxfv2"
Jul 01 16:59:42.977539 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:42.977508    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73d415ea-62f2-4da8-8610-93fbbff6cf60-registry-tls\") pod \"image-registry-86ccb9dc9-trl2q\" (UID: \"73d415ea-62f2-4da8-8610-93fbbff6cf60\") " pod="openshift-image-registry/image-registry-86ccb9dc9-trl2q"
Jul 01 16:59:42.977693 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:42.977559    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6867bb01-cbf2-4ced-8a1e-a6fa15bfa8d7-metrics-tls\") pod \"dns-default-wr6x6\" (UID: \"6867bb01-cbf2-4ced-8a1e-a6fa15bfa8d7\") " pod="openshift-dns/dns-default-wr6x6"
Jul 01 16:59:42.977693 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:59:42.977654    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 16:59:42.977804 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:59:42.977687    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 16:59:42.977804 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:59:42.977711    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86ccb9dc9-trl2q: secret "image-registry-tls" not found
Jul 01 16:59:42.977804 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:59:42.977751    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6867bb01-cbf2-4ced-8a1e-a6fa15bfa8d7-metrics-tls podName:6867bb01-cbf2-4ced-8a1e-a6fa15bfa8d7 nodeName:}" failed. No retries permitted until 2026-07-01 17:00:46.977731182 +0000 UTC m=+164.774869561 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6867bb01-cbf2-4ced-8a1e-a6fa15bfa8d7-metrics-tls") pod "dns-default-wr6x6" (UID: "6867bb01-cbf2-4ced-8a1e-a6fa15bfa8d7") : secret "dns-default-metrics-tls" not found
Jul 01 16:59:42.977804 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:59:42.977784    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73d415ea-62f2-4da8-8610-93fbbff6cf60-registry-tls podName:73d415ea-62f2-4da8-8610-93fbbff6cf60 nodeName:}" failed. No retries permitted until 2026-07-01 17:00:46.977762369 +0000 UTC m=+164.774900771 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/73d415ea-62f2-4da8-8610-93fbbff6cf60-registry-tls") pod "image-registry-86ccb9dc9-trl2q" (UID: "73d415ea-62f2-4da8-8610-93fbbff6cf60") : secret "image-registry-tls" not found
Jul 01 16:59:42.987429 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:42.987401    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-hxfv2"
Jul 01 16:59:43.015506 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:43.015478    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-95stp"]
Jul 01 16:59:43.018901 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:59:43.018867    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0a17941_c9cd_4865_93e1_0a07014a2857.slice/crio-51d5c5ca1fdc101e9a58b8d93fd1126e55364d9bb44ab011fdcc4ca218002f0c WatchSource:0}: Error finding container 51d5c5ca1fdc101e9a58b8d93fd1126e55364d9bb44ab011fdcc4ca218002f0c: Status 404 returned error can't find the container with id 51d5c5ca1fdc101e9a58b8d93fd1126e55364d9bb44ab011fdcc4ca218002f0c
Jul 01 16:59:43.040009 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:43.039977    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:59:43.087580 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:43.087536    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-95stp" event={"ID":"c0a17941-c9cd-4865-93e1-0a07014a2857","Type":"ContainerStarted","Data":"51d5c5ca1fdc101e9a58b8d93fd1126e55364d9bb44ab011fdcc4ca218002f0c"}
Jul 01 16:59:43.126726 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:59:43.126660    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9590c1d_88ae_4902_b0ed_2ce2cf13435b.slice/crio-4033f8ef1229aa82ff80010ad8029aa51ad8f487a57c12be3b34bc7a9b331273 WatchSource:0}: Error finding container 4033f8ef1229aa82ff80010ad8029aa51ad8f487a57c12be3b34bc7a9b331273: Status 404 returned error can't find the container with id 4033f8ef1229aa82ff80010ad8029aa51ad8f487a57c12be3b34bc7a9b331273
Jul 01 16:59:43.127648 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:43.127626    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-hxfv2"]
Jul 01 16:59:43.188745 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:43.188707    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:59:44.091698 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:44.091653    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-hxfv2" event={"ID":"e9590c1d-88ae-4902-b0ed-2ce2cf13435b","Type":"ContainerStarted","Data":"4033f8ef1229aa82ff80010ad8029aa51ad8f487a57c12be3b34bc7a9b331273"}
Jul 01 16:59:45.022083 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:45.022053    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-txjpp"
Jul 01 16:59:45.096732 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:45.096702    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-hxfv2" event={"ID":"e9590c1d-88ae-4902-b0ed-2ce2cf13435b","Type":"ContainerStarted","Data":"b3311f285a27e5af014436fc2ef940b46bec2d948069da90df097b4ea1998317"}
Jul 01 16:59:45.121109 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:45.121058    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-hxfv2" podStartSLOduration=1.934700156 podStartE2EDuration="3.121039943s" totalImagesPullingTime="1.186339787s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 16:59:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 16:59:45.120669408 +0000 UTC m=+102.917807800" watchObservedRunningTime="2026-07-01 16:59:45.121039943 +0000 UTC m=+102.918178344"
Jul 01 16:59:45.278420 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:45.278297    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:59:46.056196 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:46.056161    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-jncl2"]
Jul 01 16:59:46.075298 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:46.075268    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jncl2"
Jul 01 16:59:46.081230 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:46.081092    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 01 16:59:46.081230 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:46.081117    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 01 16:59:46.081607 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:46.081581    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 01 16:59:46.081674 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:46.081581    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-57ng4\""
Jul 01 16:59:46.081725 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:46.081587    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 01 16:59:46.101145 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:46.101108    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-95stp" event={"ID":"c0a17941-c9cd-4865-93e1-0a07014a2857","Type":"ContainerStarted","Data":"42aed04dd86b3c67bcabd1681ec01e94884fe89bc3bfd5435a6c4423ffec2d2d"}
Jul 01 16:59:46.140361 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:46.140319    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jncl2"]
Jul 01 16:59:46.201788 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:46.201762    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e0f013d5-f637-41e2-8eb2-b011328fd88b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jncl2\" (UID: \"e0f013d5-f637-41e2-8eb2-b011328fd88b\") " pod="openshift-insights/insights-runtime-extractor-jncl2"
Jul 01 16:59:46.201788 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:46.201793    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e0f013d5-f637-41e2-8eb2-b011328fd88b-crio-socket\") pod \"insights-runtime-extractor-jncl2\" (UID: \"e0f013d5-f637-41e2-8eb2-b011328fd88b\") " pod="openshift-insights/insights-runtime-extractor-jncl2"
Jul 01 16:59:46.201979 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:46.201809    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d249k\" (UniqueName: \"kubernetes.io/projected/e0f013d5-f637-41e2-8eb2-b011328fd88b-kube-api-access-d249k\") pod \"insights-runtime-extractor-jncl2\" (UID: \"e0f013d5-f637-41e2-8eb2-b011328fd88b\") " pod="openshift-insights/insights-runtime-extractor-jncl2"
Jul 01 16:59:46.201979 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:46.201834    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e0f013d5-f637-41e2-8eb2-b011328fd88b-data-volume\") pod \"insights-runtime-extractor-jncl2\" (UID: \"e0f013d5-f637-41e2-8eb2-b011328fd88b\") " pod="openshift-insights/insights-runtime-extractor-jncl2"
Jul 01 16:59:46.201979 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:46.201879    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e0f013d5-f637-41e2-8eb2-b011328fd88b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jncl2\" (UID: \"e0f013d5-f637-41e2-8eb2-b011328fd88b\") " pod="openshift-insights/insights-runtime-extractor-jncl2"
Jul 01 16:59:46.268892 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:46.268839    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:59:46.272505 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:46.272461    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-95stp" podStartSLOduration=2.248999512 podStartE2EDuration="4.272446871s" totalImagesPullingTime="2.023447359s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 16:59:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 16:59:46.271523455 +0000 UTC m=+104.068661856" watchObservedRunningTime="2026-07-01 16:59:46.272446871 +0000 UTC m=+104.069585273"
Jul 01 16:59:46.303213 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:46.303178    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e0f013d5-f637-41e2-8eb2-b011328fd88b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jncl2\" (UID: \"e0f013d5-f637-41e2-8eb2-b011328fd88b\") " pod="openshift-insights/insights-runtime-extractor-jncl2"
Jul 01 16:59:46.303375 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:46.303220    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e0f013d5-f637-41e2-8eb2-b011328fd88b-crio-socket\") pod \"insights-runtime-extractor-jncl2\" (UID: \"e0f013d5-f637-41e2-8eb2-b011328fd88b\") " pod="openshift-insights/insights-runtime-extractor-jncl2"
Jul 01 16:59:46.303375 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:46.303246    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d249k\" (UniqueName: \"kubernetes.io/projected/e0f013d5-f637-41e2-8eb2-b011328fd88b-kube-api-access-d249k\") pod \"insights-runtime-extractor-jncl2\" (UID: \"e0f013d5-f637-41e2-8eb2-b011328fd88b\") " pod="openshift-insights/insights-runtime-extractor-jncl2"
Jul 01 16:59:46.303375 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:59:46.303266    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 01 16:59:46.303375 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:46.303285    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e0f013d5-f637-41e2-8eb2-b011328fd88b-data-volume\") pod \"insights-runtime-extractor-jncl2\" (UID: \"e0f013d5-f637-41e2-8eb2-b011328fd88b\") " pod="openshift-insights/insights-runtime-extractor-jncl2"
Jul 01 16:59:46.303375 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:46.303312    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e0f013d5-f637-41e2-8eb2-b011328fd88b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jncl2\" (UID: \"e0f013d5-f637-41e2-8eb2-b011328fd88b\") " pod="openshift-insights/insights-runtime-extractor-jncl2"
Jul 01 16:59:46.303375 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:59:46.303335    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e0f013d5-f637-41e2-8eb2-b011328fd88b-kube-rbac-proxy-cm podName:e0f013d5-f637-41e2-8eb2-b011328fd88b nodeName:}" failed. No retries permitted until 2026-07-01 16:59:46.803316648 +0000 UTC m=+104.600455084 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e0f013d5-f637-41e2-8eb2-b011328fd88b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jncl2" (UID: "e0f013d5-f637-41e2-8eb2-b011328fd88b") : configmap "kube-rbac-proxy" not found
Jul 01 16:59:46.303375 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:46.303361    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e0f013d5-f637-41e2-8eb2-b011328fd88b-crio-socket\") pod \"insights-runtime-extractor-jncl2\" (UID: \"e0f013d5-f637-41e2-8eb2-b011328fd88b\") " pod="openshift-insights/insights-runtime-extractor-jncl2"
Jul 01 16:59:46.303723 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:59:46.303423    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 16:59:46.303723 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:59:46.303469    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0f013d5-f637-41e2-8eb2-b011328fd88b-insights-runtime-extractor-tls podName:e0f013d5-f637-41e2-8eb2-b011328fd88b nodeName:}" failed. No retries permitted until 2026-07-01 16:59:46.803455168 +0000 UTC m=+104.600593550 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e0f013d5-f637-41e2-8eb2-b011328fd88b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jncl2" (UID: "e0f013d5-f637-41e2-8eb2-b011328fd88b") : secret "insights-runtime-extractor-tls" not found
Jul 01 16:59:46.303973 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:46.303952    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e0f013d5-f637-41e2-8eb2-b011328fd88b-data-volume\") pod \"insights-runtime-extractor-jncl2\" (UID: \"e0f013d5-f637-41e2-8eb2-b011328fd88b\") " pod="openshift-insights/insights-runtime-extractor-jncl2"
Jul 01 16:59:46.393795 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:46.393722    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d249k\" (UniqueName: \"kubernetes.io/projected/e0f013d5-f637-41e2-8eb2-b011328fd88b-kube-api-access-d249k\") pod \"insights-runtime-extractor-jncl2\" (UID: \"e0f013d5-f637-41e2-8eb2-b011328fd88b\") " pod="openshift-insights/insights-runtime-extractor-jncl2"
Jul 01 16:59:46.544437 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:46.544403    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:59:46.698929 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:46.698835    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:59:46.805376 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:46.805325    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:59:46.807286 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:46.807255    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e0f013d5-f637-41e2-8eb2-b011328fd88b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jncl2\" (UID: \"e0f013d5-f637-41e2-8eb2-b011328fd88b\") " pod="openshift-insights/insights-runtime-extractor-jncl2"
Jul 01 16:59:46.807417 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:46.807297    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e0f013d5-f637-41e2-8eb2-b011328fd88b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jncl2\" (UID: \"e0f013d5-f637-41e2-8eb2-b011328fd88b\") " pod="openshift-insights/insights-runtime-extractor-jncl2"
Jul 01 16:59:46.807417 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:59:46.807387    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 01 16:59:46.807493 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:59:46.807421    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 16:59:46.807493 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:59:46.807451    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e0f013d5-f637-41e2-8eb2-b011328fd88b-kube-rbac-proxy-cm podName:e0f013d5-f637-41e2-8eb2-b011328fd88b nodeName:}" failed. No retries permitted until 2026-07-01 16:59:47.807436377 +0000 UTC m=+105.604574755 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e0f013d5-f637-41e2-8eb2-b011328fd88b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jncl2" (UID: "e0f013d5-f637-41e2-8eb2-b011328fd88b") : configmap "kube-rbac-proxy" not found
Jul 01 16:59:46.807493 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:59:46.807467    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0f013d5-f637-41e2-8eb2-b011328fd88b-insights-runtime-extractor-tls podName:e0f013d5-f637-41e2-8eb2-b011328fd88b nodeName:}" failed. No retries permitted until 2026-07-01 16:59:47.807458935 +0000 UTC m=+105.604597314 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e0f013d5-f637-41e2-8eb2-b011328fd88b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jncl2" (UID: "e0f013d5-f637-41e2-8eb2-b011328fd88b") : secret "insights-runtime-extractor-tls" not found
Jul 01 16:59:47.461853 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:47.461809    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:59:47.564624 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:47.564587    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:59:47.802117 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:47.802089    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5bjn2_242fcfc2-df03-4f81-bd5c-1bf8ccf2d2d4/dns-node-resolver/0.log"
Jul 01 16:59:47.814423 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:47.814400    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e0f013d5-f637-41e2-8eb2-b011328fd88b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jncl2\" (UID: \"e0f013d5-f637-41e2-8eb2-b011328fd88b\") " pod="openshift-insights/insights-runtime-extractor-jncl2"
Jul 01 16:59:47.814524 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:47.814442    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e0f013d5-f637-41e2-8eb2-b011328fd88b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jncl2\" (UID: \"e0f013d5-f637-41e2-8eb2-b011328fd88b\") " pod="openshift-insights/insights-runtime-extractor-jncl2"
Jul 01 16:59:47.814524 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:59:47.814471    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 01 16:59:47.814602 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:59:47.814528    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e0f013d5-f637-41e2-8eb2-b011328fd88b-kube-rbac-proxy-cm podName:e0f013d5-f637-41e2-8eb2-b011328fd88b nodeName:}" failed. No retries permitted until 2026-07-01 16:59:49.814509081 +0000 UTC m=+107.611647462 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e0f013d5-f637-41e2-8eb2-b011328fd88b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jncl2" (UID: "e0f013d5-f637-41e2-8eb2-b011328fd88b") : configmap "kube-rbac-proxy" not found
Jul 01 16:59:47.814602 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:59:47.814589    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 16:59:47.814691 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:59:47.814639    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0f013d5-f637-41e2-8eb2-b011328fd88b-insights-runtime-extractor-tls podName:e0f013d5-f637-41e2-8eb2-b011328fd88b nodeName:}" failed. No retries permitted until 2026-07-01 16:59:49.814627319 +0000 UTC m=+107.611765699 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e0f013d5-f637-41e2-8eb2-b011328fd88b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jncl2" (UID: "e0f013d5-f637-41e2-8eb2-b011328fd88b") : secret "insights-runtime-extractor-tls" not found
Jul 01 16:59:48.170920 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:48.170834    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:59:48.271539 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:48.271507    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:59:48.798785 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:48.798758    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bjw8j_bf6178b8-7786-4321-9401-ca2ec4315089/node-ca/0.log"
Jul 01 16:59:48.957002 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:48.956967    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-7bb4f4bcc7-ncbgd"]
Jul 01 16:59:48.959773 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:48.959750    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-7bb4f4bcc7-ncbgd"
Jul 01 16:59:48.962550 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:48.962530    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 01 16:59:48.963125 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:48.963107    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 01 16:59:48.963208 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:48.963114    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 01 16:59:48.963978 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:48.963959    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-q4rdt\""
Jul 01 16:59:48.964096 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:48.963984    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 01 16:59:48.978880 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:48.978857    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-7bb4f4bcc7-ncbgd"]
Jul 01 16:59:49.010848 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:49.010813    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:59:49.086429 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:49.086291    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:59:49.126127 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:49.126108    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/78ef10c6-d632-49b7-8416-f1f0c2015153-signing-key\") pod \"service-ca-7bb4f4bcc7-ncbgd\" (UID: \"78ef10c6-d632-49b7-8416-f1f0c2015153\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-ncbgd"
Jul 01 16:59:49.126253 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:49.126132    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfzbj\" (UniqueName: \"kubernetes.io/projected/78ef10c6-d632-49b7-8416-f1f0c2015153-kube-api-access-rfzbj\") pod \"service-ca-7bb4f4bcc7-ncbgd\" (UID: \"78ef10c6-d632-49b7-8416-f1f0c2015153\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-ncbgd"
Jul 01 16:59:49.126253 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:49.126161    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/78ef10c6-d632-49b7-8416-f1f0c2015153-signing-cabundle\") pod \"service-ca-7bb4f4bcc7-ncbgd\" (UID: \"78ef10c6-d632-49b7-8416-f1f0c2015153\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-ncbgd"
Jul 01 16:59:49.227496 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:49.227458    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/78ef10c6-d632-49b7-8416-f1f0c2015153-signing-key\") pod \"service-ca-7bb4f4bcc7-ncbgd\" (UID: \"78ef10c6-d632-49b7-8416-f1f0c2015153\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-ncbgd"
Jul 01 16:59:49.227496 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:49.227494    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfzbj\" (UniqueName: \"kubernetes.io/projected/78ef10c6-d632-49b7-8416-f1f0c2015153-kube-api-access-rfzbj\") pod \"service-ca-7bb4f4bcc7-ncbgd\" (UID: \"78ef10c6-d632-49b7-8416-f1f0c2015153\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-ncbgd"
Jul 01 16:59:49.227702 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:49.227522    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/78ef10c6-d632-49b7-8416-f1f0c2015153-signing-cabundle\") pod \"service-ca-7bb4f4bcc7-ncbgd\" (UID: \"78ef10c6-d632-49b7-8416-f1f0c2015153\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-ncbgd"
Jul 01 16:59:49.228144 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:49.228126    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/78ef10c6-d632-49b7-8416-f1f0c2015153-signing-cabundle\") pod \"service-ca-7bb4f4bcc7-ncbgd\" (UID: \"78ef10c6-d632-49b7-8416-f1f0c2015153\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-ncbgd"
Jul 01 16:59:49.229993 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:49.229976    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/78ef10c6-d632-49b7-8416-f1f0c2015153-signing-key\") pod \"service-ca-7bb4f4bcc7-ncbgd\" (UID: \"78ef10c6-d632-49b7-8416-f1f0c2015153\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-ncbgd"
Jul 01 16:59:49.232903 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:49.232875    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:59:49.236513 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:49.236493    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfzbj\" (UniqueName: \"kubernetes.io/projected/78ef10c6-d632-49b7-8416-f1f0c2015153-kube-api-access-rfzbj\") pod \"service-ca-7bb4f4bcc7-ncbgd\" (UID: \"78ef10c6-d632-49b7-8416-f1f0c2015153\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-ncbgd"
Jul 01 16:59:49.268502 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:49.268471    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-7bb4f4bcc7-ncbgd"
Jul 01 16:59:49.303390 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:49.303325    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:59:49.395580 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:49.395551    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-7bb4f4bcc7-ncbgd"]
Jul 01 16:59:49.398510 ip-10-0-137-180 kubenswrapper[2563]: W0701 16:59:49.398485    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78ef10c6_d632_49b7_8416_f1f0c2015153.slice/crio-5a050583e7295a28398d7cec34580373e1bc8c9a672bb8548da40a587ffc4bb0 WatchSource:0}: Error finding container 5a050583e7295a28398d7cec34580373e1bc8c9a672bb8548da40a587ffc4bb0: Status 404 returned error can't find the container with id 5a050583e7295a28398d7cec34580373e1bc8c9a672bb8548da40a587ffc4bb0
Jul 01 16:59:49.799530 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:49.799495    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dm2q7_12b51164-c7a2-464b-8652-077ef495ff1f/serve-healthcheck-canary/0.log"
Jul 01 16:59:49.833267 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:49.833240    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e0f013d5-f637-41e2-8eb2-b011328fd88b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jncl2\" (UID: \"e0f013d5-f637-41e2-8eb2-b011328fd88b\") " pod="openshift-insights/insights-runtime-extractor-jncl2"
Jul 01 16:59:49.833431 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:49.833287    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e0f013d5-f637-41e2-8eb2-b011328fd88b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jncl2\" (UID: \"e0f013d5-f637-41e2-8eb2-b011328fd88b\") " pod="openshift-insights/insights-runtime-extractor-jncl2"
Jul 01 16:59:49.833431 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:59:49.833391    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 01 16:59:49.833550 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:59:49.833427    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 16:59:49.833550 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:59:49.833466    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e0f013d5-f637-41e2-8eb2-b011328fd88b-kube-rbac-proxy-cm podName:e0f013d5-f637-41e2-8eb2-b011328fd88b nodeName:}" failed. No retries permitted until 2026-07-01 16:59:53.833443917 +0000 UTC m=+111.630582307 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e0f013d5-f637-41e2-8eb2-b011328fd88b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jncl2" (UID: "e0f013d5-f637-41e2-8eb2-b011328fd88b") : configmap "kube-rbac-proxy" not found
Jul 01 16:59:49.833550 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:59:49.833487    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0f013d5-f637-41e2-8eb2-b011328fd88b-insights-runtime-extractor-tls podName:e0f013d5-f637-41e2-8eb2-b011328fd88b nodeName:}" failed. No retries permitted until 2026-07-01 16:59:53.833473786 +0000 UTC m=+111.630612165 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e0f013d5-f637-41e2-8eb2-b011328fd88b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jncl2" (UID: "e0f013d5-f637-41e2-8eb2-b011328fd88b") : secret "insights-runtime-extractor-tls" not found
Jul 01 16:59:50.113536 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:50.113440    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-7bb4f4bcc7-ncbgd" event={"ID":"78ef10c6-d632-49b7-8416-f1f0c2015153","Type":"ContainerStarted","Data":"ba610d148abe0483029598968bb466acfe000ae290e8ffc077c15b29754ad6d0"}
Jul 01 16:59:50.113536 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:50.113498    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-7bb4f4bcc7-ncbgd" event={"ID":"78ef10c6-d632-49b7-8416-f1f0c2015153","Type":"ContainerStarted","Data":"5a050583e7295a28398d7cec34580373e1bc8c9a672bb8548da40a587ffc4bb0"}
Jul 01 16:59:50.159979 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:50.159926    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-7bb4f4bcc7-ncbgd" podStartSLOduration=2.159910034 podStartE2EDuration="2.159910034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 16:59:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 16:59:50.15936587 +0000 UTC m=+107.956504288" watchObservedRunningTime="2026-07-01 16:59:50.159910034 +0000 UTC m=+107.957048434"
Jul 01 16:59:51.317052 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:51.317001    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:59:51.422703 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:51.422663    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:59:51.991901 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:51.991869    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:59:52.379720 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:52.379631    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:59:52.795919 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:52.795885    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:59:53.236439 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:53.236404    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:59:53.408808 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:53.408776    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:59:53.749974 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:53.749934    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:59:53.866509 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:53.866478    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e0f013d5-f637-41e2-8eb2-b011328fd88b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jncl2\" (UID: \"e0f013d5-f637-41e2-8eb2-b011328fd88b\") " pod="openshift-insights/insights-runtime-extractor-jncl2"
Jul 01 16:59:53.866701 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:53.866525    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e0f013d5-f637-41e2-8eb2-b011328fd88b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jncl2\" (UID: \"e0f013d5-f637-41e2-8eb2-b011328fd88b\") " pod="openshift-insights/insights-runtime-extractor-jncl2"
Jul 01 16:59:53.866701 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:59:53.866618    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 01 16:59:53.866701 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:59:53.866635    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 16:59:53.866701 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:59:53.866696    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e0f013d5-f637-41e2-8eb2-b011328fd88b-kube-rbac-proxy-cm podName:e0f013d5-f637-41e2-8eb2-b011328fd88b nodeName:}" failed. No retries permitted until 2026-07-01 17:00:01.866676969 +0000 UTC m=+119.663815362 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e0f013d5-f637-41e2-8eb2-b011328fd88b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jncl2" (UID: "e0f013d5-f637-41e2-8eb2-b011328fd88b") : configmap "kube-rbac-proxy" not found
Jul 01 16:59:53.866858 ip-10-0-137-180 kubenswrapper[2563]: E0701 16:59:53.866718    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0f013d5-f637-41e2-8eb2-b011328fd88b-insights-runtime-extractor-tls podName:e0f013d5-f637-41e2-8eb2-b011328fd88b nodeName:}" failed. No retries permitted until 2026-07-01 17:00:01.86670866 +0000 UTC m=+119.663847043 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e0f013d5-f637-41e2-8eb2-b011328fd88b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jncl2" (UID: "e0f013d5-f637-41e2-8eb2-b011328fd88b") : secret "insights-runtime-extractor-tls" not found
Jul 01 16:59:54.209070 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:54.208978    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:59:54.430878 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:54.430845    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:59:54.632197 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:54.632163    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:59:54.985843 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:54.985808    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:59:55.205165 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:55.205129    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:59:55.353147 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:55.353062    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:59:55.736564 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:55.736531    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:59:55.965387 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:55.965328    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:59:56.431673 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:56.431637    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:59:57.028991 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:57.028948    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:59:57.221024 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:57.220987    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:59:57.579806 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:57.579768    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:59:57.803468 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:57.803430    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:59:58.434553 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:58.434513    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:59:58.960221 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:58.960172    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:59:59.100874 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:59.100840    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:59:59.258895 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:59.258854    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:59:59.381256 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:59.381214    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 16:59:59.574747 ip-10-0-137-180 kubenswrapper[2563]: I0701 16:59:59.574650    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:00:00.574120 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:00.574086    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:00:00.823927 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:00.823889    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:00:01.139644 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:01.139602    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:00:01.495600 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:01.495569    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:00:01.671649 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:01.671593    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:00:01.915651 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:01.915562    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:00:01.929831 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:01.929806    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e0f013d5-f637-41e2-8eb2-b011328fd88b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jncl2\" (UID: \"e0f013d5-f637-41e2-8eb2-b011328fd88b\") " pod="openshift-insights/insights-runtime-extractor-jncl2"
Jul 01 17:00:01.929919 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:01.929846    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e0f013d5-f637-41e2-8eb2-b011328fd88b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jncl2\" (UID: \"e0f013d5-f637-41e2-8eb2-b011328fd88b\") " pod="openshift-insights/insights-runtime-extractor-jncl2"
Jul 01 17:00:01.930429 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:01.930408    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e0f013d5-f637-41e2-8eb2-b011328fd88b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jncl2\" (UID: \"e0f013d5-f637-41e2-8eb2-b011328fd88b\") " pod="openshift-insights/insights-runtime-extractor-jncl2"
Jul 01 17:00:01.932070 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:01.932049    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e0f013d5-f637-41e2-8eb2-b011328fd88b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jncl2\" (UID: \"e0f013d5-f637-41e2-8eb2-b011328fd88b\") " pod="openshift-insights/insights-runtime-extractor-jncl2"
Jul 01 17:00:01.984101 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:01.984070    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jncl2"
Jul 01 17:00:02.145954 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:02.145916    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jncl2"]
Jul 01 17:00:02.149237 ip-10-0-137-180 kubenswrapper[2563]: W0701 17:00:02.149200    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0f013d5_f637_41e2_8eb2_b011328fd88b.slice/crio-102bbb8b1727ef2ed3054c99f609b2f97d57499c846ca541a1b98cec09b599fb WatchSource:0}: Error finding container 102bbb8b1727ef2ed3054c99f609b2f97d57499c846ca541a1b98cec09b599fb: Status 404 returned error can't find the container with id 102bbb8b1727ef2ed3054c99f609b2f97d57499c846ca541a1b98cec09b599fb
Jul 01 17:00:02.406410 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:02.406377    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:00:03.151550 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:03.151511    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jncl2" event={"ID":"e0f013d5-f637-41e2-8eb2-b011328fd88b","Type":"ContainerStarted","Data":"102bbb8b1727ef2ed3054c99f609b2f97d57499c846ca541a1b98cec09b599fb"}
Jul 01 17:00:07.168316 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:07.168280    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jncl2" event={"ID":"e0f013d5-f637-41e2-8eb2-b011328fd88b","Type":"ContainerStarted","Data":"852902c5ebc5907195bee1a5764debfe689227e06dc70ad733f2d988d7e9ece5"}
Jul 01 17:00:07.596869 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:07.596833    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:00:09.158804 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:09.158767    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:00:09.178173 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:09.178138    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jncl2" event={"ID":"e0f013d5-f637-41e2-8eb2-b011328fd88b","Type":"ContainerStarted","Data":"036c71e10ea6342805eee9234771bb8d53ebd74549b7ad4d8a97a228d4192862"}
Jul 01 17:00:09.282073 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:09.282036    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:00:09.398486 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:09.398430    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:00:09.679357 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:09.679292    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:00:09.811627 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:09.811573    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:00:10.080013 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:10.079976    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:00:11.100607 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:11.100576    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:00:11.186145 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:11.186118    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jncl2" event={"ID":"e0f013d5-f637-41e2-8eb2-b011328fd88b","Type":"ContainerStarted","Data":"32b4844aa78e05ddd7e43c0cc9c0f317ddbd9bcc21127f95c04353a479c496cd"}
Jul 01 17:00:11.200511 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:11.200478    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:00:11.216258 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:11.216209    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-jncl2" podStartSLOduration=17.359596957 podStartE2EDuration="26.216194844s" totalImagesPullingTime="8.856597887s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 16:59:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:00:11.214911858 +0000 UTC m=+129.012050274" watchObservedRunningTime="2026-07-01 17:00:11.216194844 +0000 UTC m=+129.013333292"
Jul 01 17:00:11.635875 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:11.635843    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:00:11.979278 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:11.979248    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:00:12.517583 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:12.517542    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/899ca2bf-8e98-4def-a2f9-ace75d3170a0-metrics-certs\") pod \"network-metrics-daemon-fp8j8\" (UID: \"899ca2bf-8e98-4def-a2f9-ace75d3170a0\") " pod="openshift-multus/network-metrics-daemon-fp8j8"
Jul 01 17:00:12.519960 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:12.519943    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/899ca2bf-8e98-4def-a2f9-ace75d3170a0-metrics-certs\") pod \"network-metrics-daemon-fp8j8\" (UID: \"899ca2bf-8e98-4def-a2f9-ace75d3170a0\") " pod="openshift-multus/network-metrics-daemon-fp8j8"
Jul 01 17:00:12.768120 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:12.768028    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:00:12.806363 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:12.806321    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6c472\""
Jul 01 17:00:12.813947 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:12.813925    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fp8j8"
Jul 01 17:00:12.949042 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:12.949008    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fp8j8"]
Jul 01 17:00:12.952089 ip-10-0-137-180 kubenswrapper[2563]: W0701 17:00:12.952056    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod899ca2bf_8e98_4def_a2f9_ace75d3170a0.slice/crio-b142000f8a0a63443765072c146d9d6051c8fe168792e144a373c4f7dc9393a8 WatchSource:0}: Error finding container b142000f8a0a63443765072c146d9d6051c8fe168792e144a373c4f7dc9393a8: Status 404 returned error can't find the container with id b142000f8a0a63443765072c146d9d6051c8fe168792e144a373c4f7dc9393a8
Jul 01 17:00:13.022441 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:13.022357    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:00:13.146591 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:13.146562    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6ff6b8c5d8-4vqh8"]
Jul 01 17:00:13.150792 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:13.150776    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6ff6b8c5d8-4vqh8"
Jul 01 17:00:13.155843 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:13.155818    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 01 17:00:13.155843 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:13.155822    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 01 17:00:13.156022 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:13.155840    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 01 17:00:13.156147 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:13.155824    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 01 17:00:13.156714 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:13.156512    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 01 17:00:13.156714 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:13.156548    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 01 17:00:13.162152 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:13.162137    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 01 17:00:13.194222 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:13.194188    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fp8j8" event={"ID":"899ca2bf-8e98-4def-a2f9-ace75d3170a0","Type":"ContainerStarted","Data":"b142000f8a0a63443765072c146d9d6051c8fe168792e144a373c4f7dc9393a8"}
Jul 01 17:00:13.206259 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:13.206223    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6ff6b8c5d8-4vqh8"]
Jul 01 17:00:13.223572 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:13.223520    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/24f5c135-1d81-40b1-86cd-2ce9d0871ea9-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6ff6b8c5d8-4vqh8\" (UID: \"24f5c135-1d81-40b1-86cd-2ce9d0871ea9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6ff6b8c5d8-4vqh8"
Jul 01 17:00:13.223738 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:13.223635    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7gqc\" (UniqueName: \"kubernetes.io/projected/24f5c135-1d81-40b1-86cd-2ce9d0871ea9-kube-api-access-v7gqc\") pod \"cluster-proxy-proxy-agent-6ff6b8c5d8-4vqh8\" (UID: \"24f5c135-1d81-40b1-86cd-2ce9d0871ea9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6ff6b8c5d8-4vqh8"
Jul 01 17:00:13.223738 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:13.223670    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/24f5c135-1d81-40b1-86cd-2ce9d0871ea9-hub\") pod \"cluster-proxy-proxy-agent-6ff6b8c5d8-4vqh8\" (UID: \"24f5c135-1d81-40b1-86cd-2ce9d0871ea9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6ff6b8c5d8-4vqh8"
Jul 01 17:00:13.223738 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:13.223688    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/24f5c135-1d81-40b1-86cd-2ce9d0871ea9-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6ff6b8c5d8-4vqh8\" (UID: \"24f5c135-1d81-40b1-86cd-2ce9d0871ea9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6ff6b8c5d8-4vqh8"
Jul 01 17:00:13.223738 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:13.223732    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/24f5c135-1d81-40b1-86cd-2ce9d0871ea9-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6ff6b8c5d8-4vqh8\" (UID: \"24f5c135-1d81-40b1-86cd-2ce9d0871ea9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6ff6b8c5d8-4vqh8"
Jul 01 17:00:13.223920 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:13.223759    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/24f5c135-1d81-40b1-86cd-2ce9d0871ea9-ca\") pod \"cluster-proxy-proxy-agent-6ff6b8c5d8-4vqh8\" (UID: \"24f5c135-1d81-40b1-86cd-2ce9d0871ea9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6ff6b8c5d8-4vqh8"
Jul 01 17:00:13.324583 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:13.324502    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/24f5c135-1d81-40b1-86cd-2ce9d0871ea9-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6ff6b8c5d8-4vqh8\" (UID: \"24f5c135-1d81-40b1-86cd-2ce9d0871ea9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6ff6b8c5d8-4vqh8"
Jul 01 17:00:13.324583 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:13.324553    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/24f5c135-1d81-40b1-86cd-2ce9d0871ea9-ca\") pod \"cluster-proxy-proxy-agent-6ff6b8c5d8-4vqh8\" (UID: \"24f5c135-1d81-40b1-86cd-2ce9d0871ea9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6ff6b8c5d8-4vqh8"
Jul 01 17:00:13.324801 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:13.324588    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/24f5c135-1d81-40b1-86cd-2ce9d0871ea9-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6ff6b8c5d8-4vqh8\" (UID: \"24f5c135-1d81-40b1-86cd-2ce9d0871ea9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6ff6b8c5d8-4vqh8"
Jul 01 17:00:13.324801 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:13.324642    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7gqc\" (UniqueName: \"kubernetes.io/projected/24f5c135-1d81-40b1-86cd-2ce9d0871ea9-kube-api-access-v7gqc\") pod \"cluster-proxy-proxy-agent-6ff6b8c5d8-4vqh8\" (UID: \"24f5c135-1d81-40b1-86cd-2ce9d0871ea9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6ff6b8c5d8-4vqh8"
Jul 01 17:00:13.324801 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:13.324684    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/24f5c135-1d81-40b1-86cd-2ce9d0871ea9-hub\") pod \"cluster-proxy-proxy-agent-6ff6b8c5d8-4vqh8\" (UID: \"24f5c135-1d81-40b1-86cd-2ce9d0871ea9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6ff6b8c5d8-4vqh8"
Jul 01 17:00:13.324801 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:13.324711    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/24f5c135-1d81-40b1-86cd-2ce9d0871ea9-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6ff6b8c5d8-4vqh8\" (UID: \"24f5c135-1d81-40b1-86cd-2ce9d0871ea9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6ff6b8c5d8-4vqh8"
Jul 01 17:00:13.325706 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:13.325672    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/24f5c135-1d81-40b1-86cd-2ce9d0871ea9-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6ff6b8c5d8-4vqh8\" (UID: \"24f5c135-1d81-40b1-86cd-2ce9d0871ea9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6ff6b8c5d8-4vqh8"
Jul 01 17:00:13.327265 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:13.327232    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/24f5c135-1d81-40b1-86cd-2ce9d0871ea9-ca\") pod \"cluster-proxy-proxy-agent-6ff6b8c5d8-4vqh8\" (UID: \"24f5c135-1d81-40b1-86cd-2ce9d0871ea9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6ff6b8c5d8-4vqh8"
Jul 01 17:00:13.327381 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:13.327313    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/24f5c135-1d81-40b1-86cd-2ce9d0871ea9-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6ff6b8c5d8-4vqh8\" (UID: \"24f5c135-1d81-40b1-86cd-2ce9d0871ea9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6ff6b8c5d8-4vqh8"
Jul 01 17:00:13.327427 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:13.327394    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/24f5c135-1d81-40b1-86cd-2ce9d0871ea9-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6ff6b8c5d8-4vqh8\" (UID: \"24f5c135-1d81-40b1-86cd-2ce9d0871ea9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6ff6b8c5d8-4vqh8"
Jul 01 17:00:13.327465 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:13.327430    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/24f5c135-1d81-40b1-86cd-2ce9d0871ea9-hub\") pod \"cluster-proxy-proxy-agent-6ff6b8c5d8-4vqh8\" (UID: \"24f5c135-1d81-40b1-86cd-2ce9d0871ea9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6ff6b8c5d8-4vqh8"
Jul 01 17:00:13.334959 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:13.334928    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:00:13.345270 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:13.345251    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7gqc\" (UniqueName: \"kubernetes.io/projected/24f5c135-1d81-40b1-86cd-2ce9d0871ea9-kube-api-access-v7gqc\") pod \"cluster-proxy-proxy-agent-6ff6b8c5d8-4vqh8\" (UID: \"24f5c135-1d81-40b1-86cd-2ce9d0871ea9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6ff6b8c5d8-4vqh8"
Jul 01 17:00:13.470514 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:13.470472    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6ff6b8c5d8-4vqh8"
Jul 01 17:00:13.567923 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:13.566828    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:00:13.676741 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:13.676664    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6ff6b8c5d8-4vqh8"]
Jul 01 17:00:13.854877 ip-10-0-137-180 kubenswrapper[2563]: W0701 17:00:13.854844    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod24f5c135_1d81_40b1_86cd_2ce9d0871ea9.slice/crio-87d3c699386c92fec14e96b031b17283caa4452f655e0b53018a198c2d92114e WatchSource:0}: Error finding container 87d3c699386c92fec14e96b031b17283caa4452f655e0b53018a198c2d92114e: Status 404 returned error can't find the container with id 87d3c699386c92fec14e96b031b17283caa4452f655e0b53018a198c2d92114e
Jul 01 17:00:14.198919 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:14.198832    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fp8j8" event={"ID":"899ca2bf-8e98-4def-a2f9-ace75d3170a0","Type":"ContainerStarted","Data":"1e5e7924c34290ad96fec216f3e1ce94397bfec4836ad349df0e797791168d2f"}
Jul 01 17:00:14.198919 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:14.198891    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fp8j8" event={"ID":"899ca2bf-8e98-4def-a2f9-ace75d3170a0","Type":"ContainerStarted","Data":"b4873060cb2898e5516a6ca2f177049417e129318aa7a9f2ed4ab875761e5a9d"}
Jul 01 17:00:14.200008 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:14.199989    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6ff6b8c5d8-4vqh8" event={"ID":"24f5c135-1d81-40b1-86cd-2ce9d0871ea9","Type":"ContainerStarted","Data":"87d3c699386c92fec14e96b031b17283caa4452f655e0b53018a198c2d92114e"}
Jul 01 17:00:14.301417 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:14.301335    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-fp8j8" podStartSLOduration=131.354265688 podStartE2EDuration="2m12.301319099s" totalImagesPullingTime="947.053411ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 16:58:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:00:14.30129841 +0000 UTC m=+132.098436832" watchObservedRunningTime="2026-07-01 17:00:14.301319099 +0000 UTC m=+132.098457500"
Jul 01 17:00:14.879406 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:14.879370    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:00:15.111396 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:15.111363    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:00:15.980859 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:15.980817    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:00:16.232193 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:16.232005    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:00:16.460124 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:16.460074    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:00:16.624035 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:16.623929    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:00:16.965044 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:16.964957    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:00:19.221459 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:19.221424    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6ff6b8c5d8-4vqh8" event={"ID":"24f5c135-1d81-40b1-86cd-2ce9d0871ea9","Type":"ContainerStarted","Data":"6fd6e85f3930bc511ed5a5692fbfbaaf1852b150f0c2514e4c35120d0fa0f89c"}
Jul 01 17:00:19.221792 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:19.221468    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6ff6b8c5d8-4vqh8" event={"ID":"24f5c135-1d81-40b1-86cd-2ce9d0871ea9","Type":"ContainerStarted","Data":"75063d0c89683ae6d3d6badc452e1e985a3d055dbff09b1d4526c486b5d65cdb"}
Jul 01 17:00:19.221792 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:19.221481    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6ff6b8c5d8-4vqh8" event={"ID":"24f5c135-1d81-40b1-86cd-2ce9d0871ea9","Type":"ContainerStarted","Data":"b32d8268013b24abb4fb5bffd3ceebba84d414d7164b7e206e68e0ba87823d48"}
Jul 01 17:00:19.251747 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:19.251686    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6ff6b8c5d8-4vqh8" podStartSLOduration=1.910850561 podStartE2EDuration="6.251665412s" totalImagesPullingTime="4.340814851s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:00:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:00:19.251374323 +0000 UTC m=+137.048512723" watchObservedRunningTime="2026-07-01 17:00:19.251665412 +0000 UTC m=+137.048803813"
Jul 01 17:00:21.023272 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:21.023231    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:00:21.211781 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:21.211740    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:00:22.079142 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:22.079098    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:00:22.853292 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:22.853259    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:00:23.015921 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:23.015877    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:00:24.174028 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:24.173995    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:00:24.608838 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:24.608804    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:00:24.806248 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:24.806211    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:00:24.924021 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:24.923931    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-74b92"]
Jul 01 17:00:24.927224 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:24.927208    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-74b92"
Jul 01 17:00:24.938476 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:24.938457    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 01 17:00:24.938959 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:24.938937    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 01 17:00:24.939252 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:24.939236    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-l7mm4\""
Jul 01 17:00:24.940386 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:24.940330    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 01 17:00:24.940386 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:24.940376    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 01 17:00:24.940559 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:24.940477    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 01 17:00:24.941694 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:24.941680    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 01 17:00:25.016946 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:25.016914    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/03a13c17-46c7-4f2d-9c1e-0282a2b5b379-node-exporter-textfile\") pod \"node-exporter-74b92\" (UID: \"03a13c17-46c7-4f2d-9c1e-0282a2b5b379\") " pod="openshift-monitoring/node-exporter-74b92"
Jul 01 17:00:25.016946 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:25.016946    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/03a13c17-46c7-4f2d-9c1e-0282a2b5b379-metrics-client-ca\") pod \"node-exporter-74b92\" (UID: \"03a13c17-46c7-4f2d-9c1e-0282a2b5b379\") " pod="openshift-monitoring/node-exporter-74b92"
Jul 01 17:00:25.017127 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:25.016970    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/03a13c17-46c7-4f2d-9c1e-0282a2b5b379-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-74b92\" (UID: \"03a13c17-46c7-4f2d-9c1e-0282a2b5b379\") " pod="openshift-monitoring/node-exporter-74b92"
Jul 01 17:00:25.017127 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:25.017042    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/03a13c17-46c7-4f2d-9c1e-0282a2b5b379-node-exporter-accelerators-collector-config\") pod \"node-exporter-74b92\" (UID: \"03a13c17-46c7-4f2d-9c1e-0282a2b5b379\") " pod="openshift-monitoring/node-exporter-74b92"
Jul 01 17:00:25.017127 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:25.017093    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/03a13c17-46c7-4f2d-9c1e-0282a2b5b379-sys\") pod \"node-exporter-74b92\" (UID: \"03a13c17-46c7-4f2d-9c1e-0282a2b5b379\") " pod="openshift-monitoring/node-exporter-74b92"
Jul 01 17:00:25.017127 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:25.017109    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr9h6\" (UniqueName: \"kubernetes.io/projected/03a13c17-46c7-4f2d-9c1e-0282a2b5b379-kube-api-access-cr9h6\") pod \"node-exporter-74b92\" (UID: \"03a13c17-46c7-4f2d-9c1e-0282a2b5b379\") " pod="openshift-monitoring/node-exporter-74b92"
Jul 01 17:00:25.017251 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:25.017137    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/03a13c17-46c7-4f2d-9c1e-0282a2b5b379-root\") pod \"node-exporter-74b92\" (UID: \"03a13c17-46c7-4f2d-9c1e-0282a2b5b379\") " pod="openshift-monitoring/node-exporter-74b92"
Jul 01 17:00:25.017251 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:25.017161    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/03a13c17-46c7-4f2d-9c1e-0282a2b5b379-node-exporter-wtmp\") pod \"node-exporter-74b92\" (UID: \"03a13c17-46c7-4f2d-9c1e-0282a2b5b379\") " pod="openshift-monitoring/node-exporter-74b92"
Jul 01 17:00:25.017251 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:25.017182    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/03a13c17-46c7-4f2d-9c1e-0282a2b5b379-node-exporter-tls\") pod \"node-exporter-74b92\" (UID: \"03a13c17-46c7-4f2d-9c1e-0282a2b5b379\") " pod="openshift-monitoring/node-exporter-74b92"
Jul 01 17:00:25.033768 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:25.033740    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:00:25.118429 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:25.118397    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/03a13c17-46c7-4f2d-9c1e-0282a2b5b379-sys\") pod \"node-exporter-74b92\" (UID: \"03a13c17-46c7-4f2d-9c1e-0282a2b5b379\") " pod="openshift-monitoring/node-exporter-74b92"
Jul 01 17:00:25.118429 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:25.118432    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr9h6\" (UniqueName: \"kubernetes.io/projected/03a13c17-46c7-4f2d-9c1e-0282a2b5b379-kube-api-access-cr9h6\") pod \"node-exporter-74b92\" (UID: \"03a13c17-46c7-4f2d-9c1e-0282a2b5b379\") " pod="openshift-monitoring/node-exporter-74b92"
Jul 01 17:00:25.118665 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:25.118456    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/03a13c17-46c7-4f2d-9c1e-0282a2b5b379-root\") pod \"node-exporter-74b92\" (UID: \"03a13c17-46c7-4f2d-9c1e-0282a2b5b379\") " pod="openshift-monitoring/node-exporter-74b92"
Jul 01 17:00:25.118665 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:25.118488    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/03a13c17-46c7-4f2d-9c1e-0282a2b5b379-node-exporter-wtmp\") pod \"node-exporter-74b92\" (UID: \"03a13c17-46c7-4f2d-9c1e-0282a2b5b379\") " pod="openshift-monitoring/node-exporter-74b92"
Jul 01 17:00:25.118665 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:25.118510    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/03a13c17-46c7-4f2d-9c1e-0282a2b5b379-sys\") pod \"node-exporter-74b92\" (UID: \"03a13c17-46c7-4f2d-9c1e-0282a2b5b379\") " pod="openshift-monitoring/node-exporter-74b92"
Jul 01 17:00:25.118665 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:25.118519    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/03a13c17-46c7-4f2d-9c1e-0282a2b5b379-node-exporter-tls\") pod \"node-exporter-74b92\" (UID: \"03a13c17-46c7-4f2d-9c1e-0282a2b5b379\") " pod="openshift-monitoring/node-exporter-74b92"
Jul 01 17:00:25.118665 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:25.118590    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/03a13c17-46c7-4f2d-9c1e-0282a2b5b379-root\") pod \"node-exporter-74b92\" (UID: \"03a13c17-46c7-4f2d-9c1e-0282a2b5b379\") " pod="openshift-monitoring/node-exporter-74b92"
Jul 01 17:00:25.118665 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:25.118605    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/03a13c17-46c7-4f2d-9c1e-0282a2b5b379-node-exporter-textfile\") pod \"node-exporter-74b92\" (UID: \"03a13c17-46c7-4f2d-9c1e-0282a2b5b379\") " pod="openshift-monitoring/node-exporter-74b92"
Jul 01 17:00:25.118665 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:25.118629    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/03a13c17-46c7-4f2d-9c1e-0282a2b5b379-metrics-client-ca\") pod \"node-exporter-74b92\" (UID: \"03a13c17-46c7-4f2d-9c1e-0282a2b5b379\") " pod="openshift-monitoring/node-exporter-74b92"
Jul 01 17:00:25.118665 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:25.118648    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/03a13c17-46c7-4f2d-9c1e-0282a2b5b379-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-74b92\" (UID: \"03a13c17-46c7-4f2d-9c1e-0282a2b5b379\") " pod="openshift-monitoring/node-exporter-74b92"
Jul 01 17:00:25.119045 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:25.118706    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/03a13c17-46c7-4f2d-9c1e-0282a2b5b379-node-exporter-accelerators-collector-config\") pod \"node-exporter-74b92\" (UID: \"03a13c17-46c7-4f2d-9c1e-0282a2b5b379\") " pod="openshift-monitoring/node-exporter-74b92"
Jul 01 17:00:25.119045 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:25.118703    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/03a13c17-46c7-4f2d-9c1e-0282a2b5b379-node-exporter-wtmp\") pod \"node-exporter-74b92\" (UID: \"03a13c17-46c7-4f2d-9c1e-0282a2b5b379\") " pod="openshift-monitoring/node-exporter-74b92"
Jul 01 17:00:25.119045 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:25.119022    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/03a13c17-46c7-4f2d-9c1e-0282a2b5b379-node-exporter-textfile\") pod \"node-exporter-74b92\" (UID: \"03a13c17-46c7-4f2d-9c1e-0282a2b5b379\") " pod="openshift-monitoring/node-exporter-74b92"
Jul 01 17:00:25.119258 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:25.119239    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/03a13c17-46c7-4f2d-9c1e-0282a2b5b379-node-exporter-accelerators-collector-config\") pod \"node-exporter-74b92\" (UID: \"03a13c17-46c7-4f2d-9c1e-0282a2b5b379\") " pod="openshift-monitoring/node-exporter-74b92"
Jul 01 17:00:25.119557 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:25.119532    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/03a13c17-46c7-4f2d-9c1e-0282a2b5b379-metrics-client-ca\") pod \"node-exporter-74b92\" (UID: \"03a13c17-46c7-4f2d-9c1e-0282a2b5b379\") " pod="openshift-monitoring/node-exporter-74b92"
Jul 01 17:00:25.121075 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:25.121047    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/03a13c17-46c7-4f2d-9c1e-0282a2b5b379-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-74b92\" (UID: \"03a13c17-46c7-4f2d-9c1e-0282a2b5b379\") " pod="openshift-monitoring/node-exporter-74b92"
Jul 01 17:00:25.121165 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:25.121127    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/03a13c17-46c7-4f2d-9c1e-0282a2b5b379-node-exporter-tls\") pod \"node-exporter-74b92\" (UID: \"03a13c17-46c7-4f2d-9c1e-0282a2b5b379\") " pod="openshift-monitoring/node-exporter-74b92"
Jul 01 17:00:25.132926 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:25.132903    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr9h6\" (UniqueName: \"kubernetes.io/projected/03a13c17-46c7-4f2d-9c1e-0282a2b5b379-kube-api-access-cr9h6\") pod \"node-exporter-74b92\" (UID: \"03a13c17-46c7-4f2d-9c1e-0282a2b5b379\") " pod="openshift-monitoring/node-exporter-74b92"
Jul 01 17:00:25.236302 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:25.236269    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-74b92"
Jul 01 17:00:25.247107 ip-10-0-137-180 kubenswrapper[2563]: W0701 17:00:25.247077    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03a13c17_46c7_4f2d_9c1e_0282a2b5b379.slice/crio-123b4da63d850b49cc534f704a06b5053dc2614178b748ec9ea45b7182540003 WatchSource:0}: Error finding container 123b4da63d850b49cc534f704a06b5053dc2614178b748ec9ea45b7182540003: Status 404 returned error can't find the container with id 123b4da63d850b49cc534f704a06b5053dc2614178b748ec9ea45b7182540003
Jul 01 17:00:25.260875 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:25.260833    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:00:25.452018 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:25.451971    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:00:25.756895 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:25.756857    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:00:25.788256 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:25.788224    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 01 17:00:25.792200 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:25.792185    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:00:25.802224 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:25.802202    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 01 17:00:25.802333 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:25.802202    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 01 17:00:25.802333 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:25.802313    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 01 17:00:25.802462 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:25.802392    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 01 17:00:25.802462 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:25.802422    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 01 17:00:25.802462 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:25.802447    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 01 17:00:25.802462 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:25.802458    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 01 17:00:25.802622 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:25.802562    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 01 17:00:25.820375 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:25.820332    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 01 17:00:25.820799 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:25.820784    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-n87xs\""
Jul 01 17:00:25.869824 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:25.869793    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 01 17:00:25.925960 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:25.925922    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/96c32e70-bea8-41f0-8ff4-22ed01570975-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"96c32e70-bea8-41f0-8ff4-22ed01570975\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:00:25.926124 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:25.925990    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/96c32e70-bea8-41f0-8ff4-22ed01570975-web-config\") pod \"alertmanager-main-0\" (UID: \"96c32e70-bea8-41f0-8ff4-22ed01570975\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:00:25.926124 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:25.926030    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/96c32e70-bea8-41f0-8ff4-22ed01570975-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"96c32e70-bea8-41f0-8ff4-22ed01570975\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:00:25.926124 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:25.926054    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8n8z\" (UniqueName: \"kubernetes.io/projected/96c32e70-bea8-41f0-8ff4-22ed01570975-kube-api-access-p8n8z\") pod \"alertmanager-main-0\" (UID: \"96c32e70-bea8-41f0-8ff4-22ed01570975\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:00:25.926124 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:25.926083    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/96c32e70-bea8-41f0-8ff4-22ed01570975-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"96c32e70-bea8-41f0-8ff4-22ed01570975\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:00:25.926299 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:25.926158    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/96c32e70-bea8-41f0-8ff4-22ed01570975-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"96c32e70-bea8-41f0-8ff4-22ed01570975\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:00:25.926299 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:25.926199    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/96c32e70-bea8-41f0-8ff4-22ed01570975-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"96c32e70-bea8-41f0-8ff4-22ed01570975\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:00:25.926299 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:25.926259    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/96c32e70-bea8-41f0-8ff4-22ed01570975-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"96c32e70-bea8-41f0-8ff4-22ed01570975\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:00:25.926299 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:25.926288    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/96c32e70-bea8-41f0-8ff4-22ed01570975-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"96c32e70-bea8-41f0-8ff4-22ed01570975\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:00:25.926523 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:25.926369    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/96c32e70-bea8-41f0-8ff4-22ed01570975-config-out\") pod \"alertmanager-main-0\" (UID: \"96c32e70-bea8-41f0-8ff4-22ed01570975\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:00:25.926523 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:25.926432    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/96c32e70-bea8-41f0-8ff4-22ed01570975-config-volume\") pod \"alertmanager-main-0\" (UID: \"96c32e70-bea8-41f0-8ff4-22ed01570975\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:00:25.926523 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:25.926460    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/96c32e70-bea8-41f0-8ff4-22ed01570975-tls-assets\") pod \"alertmanager-main-0\" (UID: \"96c32e70-bea8-41f0-8ff4-22ed01570975\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:00:25.926523 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:25.926510    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96c32e70-bea8-41f0-8ff4-22ed01570975-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"96c32e70-bea8-41f0-8ff4-22ed01570975\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:00:26.027704 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:26.027622    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96c32e70-bea8-41f0-8ff4-22ed01570975-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"96c32e70-bea8-41f0-8ff4-22ed01570975\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:00:26.027704 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:26.027698    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/96c32e70-bea8-41f0-8ff4-22ed01570975-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"96c32e70-bea8-41f0-8ff4-22ed01570975\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:00:26.027931 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:26.027735    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/96c32e70-bea8-41f0-8ff4-22ed01570975-web-config\") pod \"alertmanager-main-0\" (UID: \"96c32e70-bea8-41f0-8ff4-22ed01570975\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:00:26.027931 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:26.027756    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/96c32e70-bea8-41f0-8ff4-22ed01570975-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"96c32e70-bea8-41f0-8ff4-22ed01570975\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:00:26.027931 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:26.027867    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8n8z\" (UniqueName: \"kubernetes.io/projected/96c32e70-bea8-41f0-8ff4-22ed01570975-kube-api-access-p8n8z\") pod \"alertmanager-main-0\" (UID: \"96c32e70-bea8-41f0-8ff4-22ed01570975\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:00:26.027931 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:26.027917    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/96c32e70-bea8-41f0-8ff4-22ed01570975-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"96c32e70-bea8-41f0-8ff4-22ed01570975\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:00:26.028137 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:26.027953    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/96c32e70-bea8-41f0-8ff4-22ed01570975-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"96c32e70-bea8-41f0-8ff4-22ed01570975\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:00:26.028137 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:26.027981    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/96c32e70-bea8-41f0-8ff4-22ed01570975-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"96c32e70-bea8-41f0-8ff4-22ed01570975\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:00:26.028137 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:26.028024    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/96c32e70-bea8-41f0-8ff4-22ed01570975-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"96c32e70-bea8-41f0-8ff4-22ed01570975\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:00:26.028137 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:26.028051    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/96c32e70-bea8-41f0-8ff4-22ed01570975-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"96c32e70-bea8-41f0-8ff4-22ed01570975\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:00:26.028137 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:26.028086    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/96c32e70-bea8-41f0-8ff4-22ed01570975-config-out\") pod \"alertmanager-main-0\" (UID: \"96c32e70-bea8-41f0-8ff4-22ed01570975\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:00:26.028137 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:26.028129    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/96c32e70-bea8-41f0-8ff4-22ed01570975-config-volume\") pod \"alertmanager-main-0\" (UID: \"96c32e70-bea8-41f0-8ff4-22ed01570975\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:00:26.028502 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:26.028150    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/96c32e70-bea8-41f0-8ff4-22ed01570975-tls-assets\") pod \"alertmanager-main-0\" (UID: \"96c32e70-bea8-41f0-8ff4-22ed01570975\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:00:26.028685 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:26.028638    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96c32e70-bea8-41f0-8ff4-22ed01570975-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"96c32e70-bea8-41f0-8ff4-22ed01570975\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:00:26.028947 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:26.028740    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/96c32e70-bea8-41f0-8ff4-22ed01570975-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"96c32e70-bea8-41f0-8ff4-22ed01570975\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:00:26.030122 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:26.030095    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/96c32e70-bea8-41f0-8ff4-22ed01570975-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"96c32e70-bea8-41f0-8ff4-22ed01570975\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:00:26.031286 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:26.031261    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/96c32e70-bea8-41f0-8ff4-22ed01570975-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"96c32e70-bea8-41f0-8ff4-22ed01570975\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:00:26.031424 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:26.031285    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/96c32e70-bea8-41f0-8ff4-22ed01570975-config-out\") pod \"alertmanager-main-0\" (UID: \"96c32e70-bea8-41f0-8ff4-22ed01570975\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:00:26.031424 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:26.031331    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/96c32e70-bea8-41f0-8ff4-22ed01570975-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"96c32e70-bea8-41f0-8ff4-22ed01570975\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:00:26.031774 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:26.031727    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/96c32e70-bea8-41f0-8ff4-22ed01570975-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"96c32e70-bea8-41f0-8ff4-22ed01570975\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:00:26.033321 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:26.033299    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/96c32e70-bea8-41f0-8ff4-22ed01570975-config-volume\") pod \"alertmanager-main-0\" (UID: \"96c32e70-bea8-41f0-8ff4-22ed01570975\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:00:26.033424 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:26.033370    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/96c32e70-bea8-41f0-8ff4-22ed01570975-web-config\") pod \"alertmanager-main-0\" (UID: \"96c32e70-bea8-41f0-8ff4-22ed01570975\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:00:26.033478 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:26.033464    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/96c32e70-bea8-41f0-8ff4-22ed01570975-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"96c32e70-bea8-41f0-8ff4-22ed01570975\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:00:26.033521 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:26.033481    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/96c32e70-bea8-41f0-8ff4-22ed01570975-tls-assets\") pod \"alertmanager-main-0\" (UID: \"96c32e70-bea8-41f0-8ff4-22ed01570975\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:00:26.033785 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:26.033757    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/96c32e70-bea8-41f0-8ff4-22ed01570975-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"96c32e70-bea8-41f0-8ff4-22ed01570975\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:00:26.041776 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:26.041752    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8n8z\" (UniqueName: \"kubernetes.io/projected/96c32e70-bea8-41f0-8ff4-22ed01570975-kube-api-access-p8n8z\") pod \"alertmanager-main-0\" (UID: \"96c32e70-bea8-41f0-8ff4-22ed01570975\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:00:26.060484 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:26.060453    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:00:26.103236 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:26.103215    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:00:26.244195 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:26.244163    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-74b92" event={"ID":"03a13c17-46c7-4f2d-9c1e-0282a2b5b379","Type":"ContainerStarted","Data":"123b4da63d850b49cc534f704a06b5053dc2614178b748ec9ea45b7182540003"}
Jul 01 17:00:26.323618 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:26.323541    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 01 17:00:26.453648 ip-10-0-137-180 kubenswrapper[2563]: W0701 17:00:26.453615    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96c32e70_bea8_41f0_8ff4_22ed01570975.slice/crio-152e15bdf685cd77e530650dbd134f3dc6fceaa3adee341ba70b3cce83805080 WatchSource:0}: Error finding container 152e15bdf685cd77e530650dbd134f3dc6fceaa3adee341ba70b3cce83805080: Status 404 returned error can't find the container with id 152e15bdf685cd77e530650dbd134f3dc6fceaa3adee341ba70b3cce83805080
Jul 01 17:00:27.038254 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:27.038217    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:00:27.248826 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:27.248790    2563 generic.go:356] "Generic (PLEG): container finished" podID="03a13c17-46c7-4f2d-9c1e-0282a2b5b379" containerID="6beed7708d1499ef9831f908e1cbced4e2f9806827641be353ccae5bd661fb05" exitCode=0
Jul 01 17:00:27.249211 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:27.248868    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-74b92" event={"ID":"03a13c17-46c7-4f2d-9c1e-0282a2b5b379","Type":"ContainerDied","Data":"6beed7708d1499ef9831f908e1cbced4e2f9806827641be353ccae5bd661fb05"}
Jul 01 17:00:27.250056 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:27.250036    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"96c32e70-bea8-41f0-8ff4-22ed01570975","Type":"ContainerStarted","Data":"152e15bdf685cd77e530650dbd134f3dc6fceaa3adee341ba70b3cce83805080"}
Jul 01 17:00:27.448398 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:27.448336    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:00:27.559253 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:27.559167    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:00:27.666078 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:27.666041    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:00:27.770995 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:27.770960    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:00:27.920283 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:27.920197    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:00:28.255477 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:28.255447    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-74b92" event={"ID":"03a13c17-46c7-4f2d-9c1e-0282a2b5b379","Type":"ContainerStarted","Data":"2d51718ec8a9a447fc03e22f1a1265adb688316a5ba9b3d0b70fb086fb54a3ec"}
Jul 01 17:00:28.255830 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:28.255527    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-74b92" event={"ID":"03a13c17-46c7-4f2d-9c1e-0282a2b5b379","Type":"ContainerStarted","Data":"e2c8abfd94ba943b18b2d9f403a873a753fbe7f9aa3f5986235c66e3ee15e99f"}
Jul 01 17:00:28.289974 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:28.289925    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-74b92" podStartSLOduration=2.9714937 podStartE2EDuration="4.289910223s" totalImagesPullingTime="1.247922087s" totalInitContainerRuntime="70.494436ms" isStatefulPod=true podCreationTimestamp="2026-07-01 17:00:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:00:28.28890615 +0000 UTC m=+146.086044578" watchObservedRunningTime="2026-07-01 17:00:28.289910223 +0000 UTC m=+146.087048628"
Jul 01 17:00:29.057213 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:29.057175    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:00:29.080825 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:29.080791    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-cd8656b8-k9ctl"]
Jul 01 17:00:29.084336 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:29.084311    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-cd8656b8-k9ctl"
Jul 01 17:00:29.086990 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:29.086971    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 01 17:00:29.087232 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:29.087211    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-kv7v2\""
Jul 01 17:00:29.095216 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:29.095197    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-cd8656b8-k9ctl"]
Jul 01 17:00:29.152600 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:29.152565    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6d4w\" (UniqueName: \"kubernetes.io/projected/77e56a5e-6149-4f54-bb78-99a00606d99f-kube-api-access-j6d4w\") pod \"monitoring-plugin-cd8656b8-k9ctl\" (UID: \"77e56a5e-6149-4f54-bb78-99a00606d99f\") " pod="openshift-monitoring/monitoring-plugin-cd8656b8-k9ctl"
Jul 01 17:00:29.152600 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:29.152604    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/77e56a5e-6149-4f54-bb78-99a00606d99f-monitoring-plugin-cert\") pod \"monitoring-plugin-cd8656b8-k9ctl\" (UID: \"77e56a5e-6149-4f54-bb78-99a00606d99f\") " pod="openshift-monitoring/monitoring-plugin-cd8656b8-k9ctl"
Jul 01 17:00:29.153882 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:29.153853    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:00:29.253922 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:29.253888    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6d4w\" (UniqueName: \"kubernetes.io/projected/77e56a5e-6149-4f54-bb78-99a00606d99f-kube-api-access-j6d4w\") pod \"monitoring-plugin-cd8656b8-k9ctl\" (UID: \"77e56a5e-6149-4f54-bb78-99a00606d99f\") " pod="openshift-monitoring/monitoring-plugin-cd8656b8-k9ctl"
Jul 01 17:00:29.254116 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:29.253929    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/77e56a5e-6149-4f54-bb78-99a00606d99f-monitoring-plugin-cert\") pod \"monitoring-plugin-cd8656b8-k9ctl\" (UID: \"77e56a5e-6149-4f54-bb78-99a00606d99f\") " pod="openshift-monitoring/monitoring-plugin-cd8656b8-k9ctl"
Jul 01 17:00:29.257157 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:29.257125    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/77e56a5e-6149-4f54-bb78-99a00606d99f-monitoring-plugin-cert\") pod \"monitoring-plugin-cd8656b8-k9ctl\" (UID: \"77e56a5e-6149-4f54-bb78-99a00606d99f\") " pod="openshift-monitoring/monitoring-plugin-cd8656b8-k9ctl"
Jul 01 17:00:29.267427 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:29.267402    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6d4w\" (UniqueName: \"kubernetes.io/projected/77e56a5e-6149-4f54-bb78-99a00606d99f-kube-api-access-j6d4w\") pod \"monitoring-plugin-cd8656b8-k9ctl\" (UID: \"77e56a5e-6149-4f54-bb78-99a00606d99f\") " pod="openshift-monitoring/monitoring-plugin-cd8656b8-k9ctl"
Jul 01 17:00:29.287806 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:29.287773    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7b98cc6cf5-r25z2"]
Jul 01 17:00:29.290869 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:29.290854    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7b98cc6cf5-r25z2"
Jul 01 17:00:29.298322 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:29.298204    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-ct3jj95m4am8u\""
Jul 01 17:00:29.298900 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:29.298534    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 01 17:00:29.298900 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:29.298625    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-v57m6\""
Jul 01 17:00:29.298900 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:29.298869    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 01 17:00:29.299152 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:29.299120    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 01 17:00:29.300240 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:29.300221    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 01 17:00:29.313189 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:29.313120    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7b98cc6cf5-r25z2"]
Jul 01 17:00:29.355123 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:29.355084    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e4977c17-0c23-43bd-9f44-89c83eb44868-metrics-server-audit-profiles\") pod \"metrics-server-7b98cc6cf5-r25z2\" (UID: \"e4977c17-0c23-43bd-9f44-89c83eb44868\") " pod="openshift-monitoring/metrics-server-7b98cc6cf5-r25z2"
Jul 01 17:00:29.355302 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:29.355139    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e4977c17-0c23-43bd-9f44-89c83eb44868-secret-metrics-server-tls\") pod \"metrics-server-7b98cc6cf5-r25z2\" (UID: \"e4977c17-0c23-43bd-9f44-89c83eb44868\") " pod="openshift-monitoring/metrics-server-7b98cc6cf5-r25z2"
Jul 01 17:00:29.355302 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:29.355283    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e4977c17-0c23-43bd-9f44-89c83eb44868-secret-metrics-server-client-certs\") pod \"metrics-server-7b98cc6cf5-r25z2\" (UID: \"e4977c17-0c23-43bd-9f44-89c83eb44868\") " pod="openshift-monitoring/metrics-server-7b98cc6cf5-r25z2"
Jul 01 17:00:29.355443 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:29.355326    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e4977c17-0c23-43bd-9f44-89c83eb44868-client-ca-bundle\") pod \"metrics-server-7b98cc6cf5-r25z2\" (UID: \"e4977c17-0c23-43bd-9f44-89c83eb44868\") " pod="openshift-monitoring/metrics-server-7b98cc6cf5-r25z2"
Jul 01 17:00:29.355500 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:29.355444    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fqvg\" (UniqueName: \"kubernetes.io/projected/e4977c17-0c23-43bd-9f44-89c83eb44868-kube-api-access-6fqvg\") pod \"metrics-server-7b98cc6cf5-r25z2\" (UID: \"e4977c17-0c23-43bd-9f44-89c83eb44868\") " pod="openshift-monitoring/metrics-server-7b98cc6cf5-r25z2"
Jul 01 17:00:29.355500 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:29.355472    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e4977c17-0c23-43bd-9f44-89c83eb44868-audit-log\") pod \"metrics-server-7b98cc6cf5-r25z2\" (UID: \"e4977c17-0c23-43bd-9f44-89c83eb44868\") " pod="openshift-monitoring/metrics-server-7b98cc6cf5-r25z2"
Jul 01 17:00:29.355649 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:29.355620    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4977c17-0c23-43bd-9f44-89c83eb44868-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7b98cc6cf5-r25z2\" (UID: \"e4977c17-0c23-43bd-9f44-89c83eb44868\") " pod="openshift-monitoring/metrics-server-7b98cc6cf5-r25z2"
Jul 01 17:00:29.396006 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:29.395968    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-cd8656b8-k9ctl"
Jul 01 17:00:29.411820 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:29.411787    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:00:29.456792 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:29.456752    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e4977c17-0c23-43bd-9f44-89c83eb44868-audit-log\") pod \"metrics-server-7b98cc6cf5-r25z2\" (UID: \"e4977c17-0c23-43bd-9f44-89c83eb44868\") " pod="openshift-monitoring/metrics-server-7b98cc6cf5-r25z2"
Jul 01 17:00:29.456975 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:29.456828    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4977c17-0c23-43bd-9f44-89c83eb44868-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7b98cc6cf5-r25z2\" (UID: \"e4977c17-0c23-43bd-9f44-89c83eb44868\") " pod="openshift-monitoring/metrics-server-7b98cc6cf5-r25z2"
Jul 01 17:00:29.456975 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:29.456877    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e4977c17-0c23-43bd-9f44-89c83eb44868-metrics-server-audit-profiles\") pod \"metrics-server-7b98cc6cf5-r25z2\" (UID: \"e4977c17-0c23-43bd-9f44-89c83eb44868\") " pod="openshift-monitoring/metrics-server-7b98cc6cf5-r25z2"
Jul 01 17:00:29.456975 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:29.456915    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e4977c17-0c23-43bd-9f44-89c83eb44868-secret-metrics-server-tls\") pod \"metrics-server-7b98cc6cf5-r25z2\" (UID: \"e4977c17-0c23-43bd-9f44-89c83eb44868\") " pod="openshift-monitoring/metrics-server-7b98cc6cf5-r25z2"
Jul 01 17:00:29.456975 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:29.456969    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e4977c17-0c23-43bd-9f44-89c83eb44868-secret-metrics-server-client-certs\") pod \"metrics-server-7b98cc6cf5-r25z2\" (UID: \"e4977c17-0c23-43bd-9f44-89c83eb44868\") " pod="openshift-monitoring/metrics-server-7b98cc6cf5-r25z2"
Jul 01 17:00:29.457184 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:29.457011    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e4977c17-0c23-43bd-9f44-89c83eb44868-client-ca-bundle\") pod \"metrics-server-7b98cc6cf5-r25z2\" (UID: \"e4977c17-0c23-43bd-9f44-89c83eb44868\") " pod="openshift-monitoring/metrics-server-7b98cc6cf5-r25z2"
Jul 01 17:00:29.457184 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:29.457068    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fqvg\" (UniqueName: \"kubernetes.io/projected/e4977c17-0c23-43bd-9f44-89c83eb44868-kube-api-access-6fqvg\") pod \"metrics-server-7b98cc6cf5-r25z2\" (UID: \"e4977c17-0c23-43bd-9f44-89c83eb44868\") " pod="openshift-monitoring/metrics-server-7b98cc6cf5-r25z2"
Jul 01 17:00:29.457255 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:29.457204    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e4977c17-0c23-43bd-9f44-89c83eb44868-audit-log\") pod \"metrics-server-7b98cc6cf5-r25z2\" (UID: \"e4977c17-0c23-43bd-9f44-89c83eb44868\") " pod="openshift-monitoring/metrics-server-7b98cc6cf5-r25z2"
Jul 01 17:00:29.458243 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:29.458219    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4977c17-0c23-43bd-9f44-89c83eb44868-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7b98cc6cf5-r25z2\" (UID: \"e4977c17-0c23-43bd-9f44-89c83eb44868\") " pod="openshift-monitoring/metrics-server-7b98cc6cf5-r25z2"
Jul 01 17:00:29.459035 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:29.459001    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e4977c17-0c23-43bd-9f44-89c83eb44868-metrics-server-audit-profiles\") pod \"metrics-server-7b98cc6cf5-r25z2\" (UID: \"e4977c17-0c23-43bd-9f44-89c83eb44868\") " pod="openshift-monitoring/metrics-server-7b98cc6cf5-r25z2"
Jul 01 17:00:29.460047 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:29.460025    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e4977c17-0c23-43bd-9f44-89c83eb44868-secret-metrics-server-client-certs\") pod \"metrics-server-7b98cc6cf5-r25z2\" (UID: \"e4977c17-0c23-43bd-9f44-89c83eb44868\") " pod="openshift-monitoring/metrics-server-7b98cc6cf5-r25z2"
Jul 01 17:00:29.460588 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:29.460567    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e4977c17-0c23-43bd-9f44-89c83eb44868-client-ca-bundle\") pod \"metrics-server-7b98cc6cf5-r25z2\" (UID: \"e4977c17-0c23-43bd-9f44-89c83eb44868\") " pod="openshift-monitoring/metrics-server-7b98cc6cf5-r25z2"
Jul 01 17:00:29.460685 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:29.460636    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e4977c17-0c23-43bd-9f44-89c83eb44868-secret-metrics-server-tls\") pod \"metrics-server-7b98cc6cf5-r25z2\" (UID: \"e4977c17-0c23-43bd-9f44-89c83eb44868\") " pod="openshift-monitoring/metrics-server-7b98cc6cf5-r25z2"
Jul 01 17:00:29.467269 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:29.467246    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fqvg\" (UniqueName: \"kubernetes.io/projected/e4977c17-0c23-43bd-9f44-89c83eb44868-kube-api-access-6fqvg\") pod \"metrics-server-7b98cc6cf5-r25z2\" (UID: \"e4977c17-0c23-43bd-9f44-89c83eb44868\") " pod="openshift-monitoring/metrics-server-7b98cc6cf5-r25z2"
Jul 01 17:00:29.601791 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:29.601767    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7b98cc6cf5-r25z2"
Jul 01 17:00:29.661981 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:29.661808    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-cd8656b8-k9ctl"]
Jul 01 17:00:29.682182 ip-10-0-137-180 kubenswrapper[2563]: W0701 17:00:29.682152    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77e56a5e_6149_4f54_bb78_99a00606d99f.slice/crio-4beaebe09c45d39eb46abd59ae1574699359038b7eac1798d26d59249dfae61f WatchSource:0}: Error finding container 4beaebe09c45d39eb46abd59ae1574699359038b7eac1798d26d59249dfae61f: Status 404 returned error can't find the container with id 4beaebe09c45d39eb46abd59ae1574699359038b7eac1798d26d59249dfae61f
Jul 01 17:00:29.749751 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:29.749718    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7b98cc6cf5-r25z2"]
Jul 01 17:00:29.756630 ip-10-0-137-180 kubenswrapper[2563]: W0701 17:00:29.756602    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4977c17_0c23_43bd_9f44_89c83eb44868.slice/crio-1462badc5d458c78c499a608a264777261f4121ab53e81965ea6548b2ae6527a WatchSource:0}: Error finding container 1462badc5d458c78c499a608a264777261f4121ab53e81965ea6548b2ae6527a: Status 404 returned error can't find the container with id 1462badc5d458c78c499a608a264777261f4121ab53e81965ea6548b2ae6527a
Jul 01 17:00:30.264851 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:30.264817    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7b98cc6cf5-r25z2" event={"ID":"e4977c17-0c23-43bd-9f44-89c83eb44868","Type":"ContainerStarted","Data":"1462badc5d458c78c499a608a264777261f4121ab53e81965ea6548b2ae6527a"}
Jul 01 17:00:30.266360 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:30.266306    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-cd8656b8-k9ctl" event={"ID":"77e56a5e-6149-4f54-bb78-99a00606d99f","Type":"ContainerStarted","Data":"4beaebe09c45d39eb46abd59ae1574699359038b7eac1798d26d59249dfae61f"}
Jul 01 17:00:30.269361 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:30.268589    2563 generic.go:356] "Generic (PLEG): container finished" podID="96c32e70-bea8-41f0-8ff4-22ed01570975" containerID="0fddaa616a1e3499b35dfc7d19a12aad5e52715c34caa2d7d050a1ea184ee7f3" exitCode=0
Jul 01 17:00:30.269361 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:30.268680    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"96c32e70-bea8-41f0-8ff4-22ed01570975","Type":"ContainerDied","Data":"0fddaa616a1e3499b35dfc7d19a12aad5e52715c34caa2d7d050a1ea184ee7f3"}
Jul 01 17:00:30.278249 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:30.278221    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:00:30.381734 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:30.381693    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:00:30.968916 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:30.968870    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:00:32.246987 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:32.246778    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:00:32.417624 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:32.417592    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:00:33.006069 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:33.006031    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:00:33.283667 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:33.283574    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7b98cc6cf5-r25z2" event={"ID":"e4977c17-0c23-43bd-9f44-89c83eb44868","Type":"ContainerStarted","Data":"3c169e997ce2eecfb77e758146a0a7caf808dd3af631c66c9567f85ba8bfd4b9"}
Jul 01 17:00:33.285066 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:33.285038    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-cd8656b8-k9ctl" event={"ID":"77e56a5e-6149-4f54-bb78-99a00606d99f","Type":"ContainerStarted","Data":"b60607a704d1c27c3d042b3f42d3531c3f51b77f77e1fd905c0d51f145dfa506"}
Jul 01 17:00:33.285203 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:33.285162    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-cd8656b8-k9ctl"
Jul 01 17:00:33.288288 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:33.288269    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"96c32e70-bea8-41f0-8ff4-22ed01570975","Type":"ContainerStarted","Data":"139aa3f5026a5668904502bc4d7b383a21e6c19f8e79e75c18e6de5c8b36aad4"}
Jul 01 17:00:33.288419 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:33.288316    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"96c32e70-bea8-41f0-8ff4-22ed01570975","Type":"ContainerStarted","Data":"ea074ff1cca8524fdca9e75ca34f71764d603f1038ee02ca27db43f29ebee801"}
Jul 01 17:00:33.288419 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:33.288356    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"96c32e70-bea8-41f0-8ff4-22ed01570975","Type":"ContainerStarted","Data":"c60eaae264c31801ef956da594ebd41b7ae7a4c9873aa7f1e7ee689dca20fe14"}
Jul 01 17:00:33.288419 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:33.288380    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"96c32e70-bea8-41f0-8ff4-22ed01570975","Type":"ContainerStarted","Data":"b39ed92f5dcb65136c7765702081de61adbde5a51de65ad8ad3d913b0be7162a"}
Jul 01 17:00:33.288419 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:33.288402    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"96c32e70-bea8-41f0-8ff4-22ed01570975","Type":"ContainerStarted","Data":"33998e56bbbf155a14799fbb93eece630ad1dbb3f062c621ca12791f2c308f39"}
Jul 01 17:00:33.290406 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:33.290387    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-cd8656b8-k9ctl"
Jul 01 17:00:33.323994 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:33.323940    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7b98cc6cf5-r25z2" podStartSLOduration=1.875247898 podStartE2EDuration="4.323924101s" totalImagesPullingTime="2.448676203s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:00:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:00:33.323777445 +0000 UTC m=+151.120915865" watchObservedRunningTime="2026-07-01 17:00:33.323924101 +0000 UTC m=+151.121062503"
Jul 01 17:00:33.348819 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:33.348785    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:00:33.361321 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:33.361273    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-cd8656b8-k9ctl" podStartSLOduration=1.842514144 podStartE2EDuration="4.36125787s" totalImagesPullingTime="2.518743726s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:00:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:00:33.36082545 +0000 UTC m=+151.157963851" watchObservedRunningTime="2026-07-01 17:00:33.36125787 +0000 UTC m=+151.158396271"
Jul 01 17:00:34.295357 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:34.295321    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"96c32e70-bea8-41f0-8ff4-22ed01570975","Type":"ContainerStarted","Data":"6ff408cef50895fb958ff2adae8564ee0046d702987b0c0e12a6bf93f01641d8"}
Jul 01 17:00:34.345929 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:34.345874    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.162912808 podStartE2EDuration="9.345854824s" totalImagesPullingTime="6.162379954s" totalInitContainerRuntime="20.562062ms" isStatefulPod=false podCreationTimestamp="2026-07-01 17:00:25 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:00:34.342944597 +0000 UTC m=+152.140082998" watchObservedRunningTime="2026-07-01 17:00:34.345854824 +0000 UTC m=+152.142993235"
Jul 01 17:00:35.711487 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:35.711452    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:00:35.962238 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:35.962156    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:00:35.974440 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:35.974407    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-86ccb9dc9-trl2q"]
Jul 01 17:00:35.974664 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:00:35.974646    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-86ccb9dc9-trl2q" podUID="73d415ea-62f2-4da8-8610-93fbbff6cf60"
Jul 01 17:00:36.150361 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:36.150313    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:00:36.291428 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:36.291396    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:00:36.302589 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:36.302560    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-86ccb9dc9-trl2q"
Jul 01 17:00:36.321292 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:36.321265    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-86ccb9dc9-trl2q"
Jul 01 17:00:36.417310 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:36.417271    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/73d415ea-62f2-4da8-8610-93fbbff6cf60-ca-trust-extracted\") pod \"73d415ea-62f2-4da8-8610-93fbbff6cf60\" (UID: \"73d415ea-62f2-4da8-8610-93fbbff6cf60\") "
Jul 01 17:00:36.417310 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:36.417309    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/73d415ea-62f2-4da8-8610-93fbbff6cf60-trusted-ca\") pod \"73d415ea-62f2-4da8-8610-93fbbff6cf60\" (UID: \"73d415ea-62f2-4da8-8610-93fbbff6cf60\") "
Jul 01 17:00:36.417557 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:36.417334    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/73d415ea-62f2-4da8-8610-93fbbff6cf60-installation-pull-secrets\") pod \"73d415ea-62f2-4da8-8610-93fbbff6cf60\" (UID: \"73d415ea-62f2-4da8-8610-93fbbff6cf60\") "
Jul 01 17:00:36.417557 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:36.417381    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/73d415ea-62f2-4da8-8610-93fbbff6cf60-registry-certificates\") pod \"73d415ea-62f2-4da8-8610-93fbbff6cf60\" (UID: \"73d415ea-62f2-4da8-8610-93fbbff6cf60\") "
Jul 01 17:00:36.417557 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:36.417496    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-snxrw\" (UniqueName: \"kubernetes.io/projected/73d415ea-62f2-4da8-8610-93fbbff6cf60-kube-api-access-snxrw\") pod \"73d415ea-62f2-4da8-8610-93fbbff6cf60\" (UID: \"73d415ea-62f2-4da8-8610-93fbbff6cf60\") "
Jul 01 17:00:36.417698 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:36.417590    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/73d415ea-62f2-4da8-8610-93fbbff6cf60-bound-sa-token\") pod \"73d415ea-62f2-4da8-8610-93fbbff6cf60\" (UID: \"73d415ea-62f2-4da8-8610-93fbbff6cf60\") "
Jul 01 17:00:36.417698 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:36.417627    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/73d415ea-62f2-4da8-8610-93fbbff6cf60-image-registry-private-configuration\") pod \"73d415ea-62f2-4da8-8610-93fbbff6cf60\" (UID: \"73d415ea-62f2-4da8-8610-93fbbff6cf60\") "
Jul 01 17:00:36.417698 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:36.417627    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73d415ea-62f2-4da8-8610-93fbbff6cf60-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "73d415ea-62f2-4da8-8610-93fbbff6cf60" (UID: "73d415ea-62f2-4da8-8610-93fbbff6cf60"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 17:00:36.417901 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:36.417737    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73d415ea-62f2-4da8-8610-93fbbff6cf60-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "73d415ea-62f2-4da8-8610-93fbbff6cf60" (UID: "73d415ea-62f2-4da8-8610-93fbbff6cf60"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:00:36.417901 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:36.417855    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73d415ea-62f2-4da8-8610-93fbbff6cf60-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "73d415ea-62f2-4da8-8610-93fbbff6cf60" (UID: "73d415ea-62f2-4da8-8610-93fbbff6cf60"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:00:36.417989 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:36.417946    2563 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/73d415ea-62f2-4da8-8610-93fbbff6cf60-ca-trust-extracted\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:00:36.417989 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:36.417963    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/73d415ea-62f2-4da8-8610-93fbbff6cf60-trusted-ca\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:00:36.417989 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:36.417975    2563 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/73d415ea-62f2-4da8-8610-93fbbff6cf60-registry-certificates\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:00:36.420027 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:36.419979    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73d415ea-62f2-4da8-8610-93fbbff6cf60-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "73d415ea-62f2-4da8-8610-93fbbff6cf60" (UID: "73d415ea-62f2-4da8-8610-93fbbff6cf60"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 17:00:36.420130 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:36.420057    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73d415ea-62f2-4da8-8610-93fbbff6cf60-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "73d415ea-62f2-4da8-8610-93fbbff6cf60" (UID: "73d415ea-62f2-4da8-8610-93fbbff6cf60"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 17:00:36.420130 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:36.420094    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73d415ea-62f2-4da8-8610-93fbbff6cf60-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "73d415ea-62f2-4da8-8610-93fbbff6cf60" (UID: "73d415ea-62f2-4da8-8610-93fbbff6cf60"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 17:00:36.420222 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:36.420125    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73d415ea-62f2-4da8-8610-93fbbff6cf60-kube-api-access-snxrw" (OuterVolumeSpecName: "kube-api-access-snxrw") pod "73d415ea-62f2-4da8-8610-93fbbff6cf60" (UID: "73d415ea-62f2-4da8-8610-93fbbff6cf60"). InnerVolumeSpecName "kube-api-access-snxrw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 17:00:36.519137 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:36.519105    2563 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/73d415ea-62f2-4da8-8610-93fbbff6cf60-installation-pull-secrets\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:00:36.519137 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:36.519131    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snxrw\" (UniqueName: \"kubernetes.io/projected/73d415ea-62f2-4da8-8610-93fbbff6cf60-kube-api-access-snxrw\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:00:36.519137 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:36.519140    2563 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/73d415ea-62f2-4da8-8610-93fbbff6cf60-bound-sa-token\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:00:36.519363 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:36.519149    2563 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/73d415ea-62f2-4da8-8610-93fbbff6cf60-image-registry-private-configuration\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:00:37.306398 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:37.306368    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-86ccb9dc9-trl2q"
Jul 01 17:00:37.349828 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:37.349793    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-86ccb9dc9-trl2q"]
Jul 01 17:00:37.354882 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:37.354859    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-86ccb9dc9-trl2q"]
Jul 01 17:00:37.426429 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:37.426406    2563 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73d415ea-62f2-4da8-8610-93fbbff6cf60-registry-tls\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:00:38.696329 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:38.696292    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73d415ea-62f2-4da8-8610-93fbbff6cf60" path="/var/lib/kubelet/pods/73d415ea-62f2-4da8-8610-93fbbff6cf60/volumes"
Jul 01 17:00:42.095820 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:42.095787    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:00:42.119757 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:00:42.119719    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-wr6x6" podUID="6867bb01-cbf2-4ced-8a1e-a6fa15bfa8d7"
Jul 01 17:00:42.323440 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:42.323414    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wr6x6"
Jul 01 17:00:43.628040 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:43.628008    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:00:43.708716 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:43.708682    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:00:44.264856 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:44.264818    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:00:45.682434 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:45.682399    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:00:45.930078 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:45.930038    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:00:46.119031 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:46.118998    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:00:47.007844 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:47.007802    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6867bb01-cbf2-4ced-8a1e-a6fa15bfa8d7-metrics-tls\") pod \"dns-default-wr6x6\" (UID: \"6867bb01-cbf2-4ced-8a1e-a6fa15bfa8d7\") " pod="openshift-dns/dns-default-wr6x6"
Jul 01 17:00:47.010277 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:47.010256    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6867bb01-cbf2-4ced-8a1e-a6fa15bfa8d7-metrics-tls\") pod \"dns-default-wr6x6\" (UID: \"6867bb01-cbf2-4ced-8a1e-a6fa15bfa8d7\") " pod="openshift-dns/dns-default-wr6x6"
Jul 01 17:00:47.128825 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:47.128788    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wm6rt\""
Jul 01 17:00:47.135228 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:47.135197    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wr6x6"
Jul 01 17:00:47.285359 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:47.285262    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wr6x6"]
Jul 01 17:00:47.288452 ip-10-0-137-180 kubenswrapper[2563]: W0701 17:00:47.288427    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6867bb01_cbf2_4ced_8a1e_a6fa15bfa8d7.slice/crio-e204371cdb14865f440c97b6302dd9e0e767507a813ead75f99715dcc1432e75 WatchSource:0}: Error finding container e204371cdb14865f440c97b6302dd9e0e767507a813ead75f99715dcc1432e75: Status 404 returned error can't find the container with id e204371cdb14865f440c97b6302dd9e0e767507a813ead75f99715dcc1432e75
Jul 01 17:00:47.342143 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:47.342104    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wr6x6" event={"ID":"6867bb01-cbf2-4ced-8a1e-a6fa15bfa8d7","Type":"ContainerStarted","Data":"e204371cdb14865f440c97b6302dd9e0e767507a813ead75f99715dcc1432e75"}
Jul 01 17:00:49.602620 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:49.602572    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7b98cc6cf5-r25z2"
Jul 01 17:00:49.603054 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:49.602647    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7b98cc6cf5-r25z2"
Jul 01 17:00:50.780643 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:50.780610    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:00:50.887975 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:50.887937    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:00:51.362633 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:51.362596    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wr6x6" event={"ID":"6867bb01-cbf2-4ced-8a1e-a6fa15bfa8d7","Type":"ContainerStarted","Data":"02abb7b26111096f03c062125a9a940a6b538f3229f5312e627b045ff7656ff2"}
Jul 01 17:00:51.362824 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:51.362671    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wr6x6" event={"ID":"6867bb01-cbf2-4ced-8a1e-a6fa15bfa8d7","Type":"ContainerStarted","Data":"7c0311738b3be19ffda85a2afb4b76fd94eac68fc7d2cbcd39ddf6f4b8c3c946"}
Jul 01 17:00:51.362824 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:51.362722    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-wr6x6"
Jul 01 17:00:51.394290 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:51.394236    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-wr6x6" podStartSLOduration=129.012829999 podStartE2EDuration="2m12.394219035s" totalImagesPullingTime="3.381389036s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 16:58:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:00:51.393675225 +0000 UTC m=+169.190813627" watchObservedRunningTime="2026-07-01 17:00:51.394219035 +0000 UTC m=+169.191357437"
Jul 01 17:00:53.315159 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:53.315112    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:00:53.384463 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:53.384420    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:00:56.382163 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:56.382077    2563 generic.go:356] "Generic (PLEG): container finished" podID="c0a17941-c9cd-4865-93e1-0a07014a2857" containerID="42aed04dd86b3c67bcabd1681ec01e94884fe89bc3bfd5435a6c4423ffec2d2d" exitCode=0
Jul 01 17:00:56.382163 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:56.382151    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-95stp" event={"ID":"c0a17941-c9cd-4865-93e1-0a07014a2857","Type":"ContainerDied","Data":"42aed04dd86b3c67bcabd1681ec01e94884fe89bc3bfd5435a6c4423ffec2d2d"}
Jul 01 17:00:56.382592 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:56.382486    2563 scope.go:117] "RemoveContainer" containerID="42aed04dd86b3c67bcabd1681ec01e94884fe89bc3bfd5435a6c4423ffec2d2d"
Jul 01 17:00:57.391861 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:57.391827    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-95stp" event={"ID":"c0a17941-c9cd-4865-93e1-0a07014a2857","Type":"ContainerStarted","Data":"c5df8e402d0cf4be1f139f4a1c0568ce6592b25bb4b9655356422ad19d02d30c"}
Jul 01 17:00:57.574707 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:57.574670    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:00:57.650038 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:00:57.649950    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:01:01.368647 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:01.368617    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-wr6x6"
Jul 01 17:01:03.464756 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:03.464721    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:01:03.639559 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:03.639522    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:01:03.728233 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:03.728201    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:01:09.607886 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:09.607852    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7b98cc6cf5-r25z2"
Jul 01 17:01:09.611753 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:09.611727    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7b98cc6cf5-r25z2"
Jul 01 17:01:10.545931 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:10.545894    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:01:10.613946 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:10.613887    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:01:12.571813 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:12.571776    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:01:12.641225 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:12.641192    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:01:40.783837 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:40.783800    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:01:40.853036 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:40.852993    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:01:45.063560 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:45.063527    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 01 17:01:45.063964 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:45.063806    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="96c32e70-bea8-41f0-8ff4-22ed01570975" containerName="alertmanager" containerID="cri-o://33998e56bbbf155a14799fbb93eece630ad1dbb3f062c621ca12791f2c308f39" gracePeriod=120
Jul 01 17:01:45.063964 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:45.063888    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="96c32e70-bea8-41f0-8ff4-22ed01570975" containerName="kube-rbac-proxy-web" containerID="cri-o://c60eaae264c31801ef956da594ebd41b7ae7a4c9873aa7f1e7ee689dca20fe14" gracePeriod=120
Jul 01 17:01:45.063964 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:45.063888    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="96c32e70-bea8-41f0-8ff4-22ed01570975" containerName="kube-rbac-proxy-metric" containerID="cri-o://139aa3f5026a5668904502bc4d7b383a21e6c19f8e79e75c18e6de5c8b36aad4" gracePeriod=120
Jul 01 17:01:45.063964 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:45.063920    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="96c32e70-bea8-41f0-8ff4-22ed01570975" containerName="kube-rbac-proxy" containerID="cri-o://ea074ff1cca8524fdca9e75ca34f71764d603f1038ee02ca27db43f29ebee801" gracePeriod=120
Jul 01 17:01:45.063964 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:45.063914    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="96c32e70-bea8-41f0-8ff4-22ed01570975" containerName="config-reloader" containerID="cri-o://b39ed92f5dcb65136c7765702081de61adbde5a51de65ad8ad3d913b0be7162a" gracePeriod=120
Jul 01 17:01:45.064241 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:45.063995    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="96c32e70-bea8-41f0-8ff4-22ed01570975" containerName="prom-label-proxy" containerID="cri-o://6ff408cef50895fb958ff2adae8564ee0046d702987b0c0e12a6bf93f01641d8" gracePeriod=120
Jul 01 17:01:45.567966 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:45.567934    2563 generic.go:356] "Generic (PLEG): container finished" podID="96c32e70-bea8-41f0-8ff4-22ed01570975" containerID="6ff408cef50895fb958ff2adae8564ee0046d702987b0c0e12a6bf93f01641d8" exitCode=0
Jul 01 17:01:45.567966 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:45.567959    2563 generic.go:356] "Generic (PLEG): container finished" podID="96c32e70-bea8-41f0-8ff4-22ed01570975" containerID="ea074ff1cca8524fdca9e75ca34f71764d603f1038ee02ca27db43f29ebee801" exitCode=0
Jul 01 17:01:45.567966 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:45.567965    2563 generic.go:356] "Generic (PLEG): container finished" podID="96c32e70-bea8-41f0-8ff4-22ed01570975" containerID="b39ed92f5dcb65136c7765702081de61adbde5a51de65ad8ad3d913b0be7162a" exitCode=0
Jul 01 17:01:45.567966 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:45.567970    2563 generic.go:356] "Generic (PLEG): container finished" podID="96c32e70-bea8-41f0-8ff4-22ed01570975" containerID="33998e56bbbf155a14799fbb93eece630ad1dbb3f062c621ca12791f2c308f39" exitCode=0
Jul 01 17:01:45.568231 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:45.568001    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"96c32e70-bea8-41f0-8ff4-22ed01570975","Type":"ContainerDied","Data":"6ff408cef50895fb958ff2adae8564ee0046d702987b0c0e12a6bf93f01641d8"}
Jul 01 17:01:45.568231 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:45.568063    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"96c32e70-bea8-41f0-8ff4-22ed01570975","Type":"ContainerDied","Data":"ea074ff1cca8524fdca9e75ca34f71764d603f1038ee02ca27db43f29ebee801"}
Jul 01 17:01:45.568231 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:45.568088    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"96c32e70-bea8-41f0-8ff4-22ed01570975","Type":"ContainerDied","Data":"b39ed92f5dcb65136c7765702081de61adbde5a51de65ad8ad3d913b0be7162a"}
Jul 01 17:01:45.568231 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:45.568111    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"96c32e70-bea8-41f0-8ff4-22ed01570975","Type":"ContainerDied","Data":"33998e56bbbf155a14799fbb93eece630ad1dbb3f062c621ca12791f2c308f39"}
Jul 01 17:01:45.990189 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:45.990161    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:01:46.083058 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.083022    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:01:46.307987 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.307957    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:01:46.478253 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.478210    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/96c32e70-bea8-41f0-8ff4-22ed01570975-secret-alertmanager-kube-rbac-proxy-web\") pod \"96c32e70-bea8-41f0-8ff4-22ed01570975\" (UID: \"96c32e70-bea8-41f0-8ff4-22ed01570975\") "
Jul 01 17:01:46.478253 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.478254    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/96c32e70-bea8-41f0-8ff4-22ed01570975-cluster-tls-config\") pod \"96c32e70-bea8-41f0-8ff4-22ed01570975\" (UID: \"96c32e70-bea8-41f0-8ff4-22ed01570975\") "
Jul 01 17:01:46.478541 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.478280    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/96c32e70-bea8-41f0-8ff4-22ed01570975-alertmanager-main-db\") pod \"96c32e70-bea8-41f0-8ff4-22ed01570975\" (UID: \"96c32e70-bea8-41f0-8ff4-22ed01570975\") "
Jul 01 17:01:46.478541 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.478303    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/96c32e70-bea8-41f0-8ff4-22ed01570975-secret-alertmanager-kube-rbac-proxy-metric\") pod \"96c32e70-bea8-41f0-8ff4-22ed01570975\" (UID: \"96c32e70-bea8-41f0-8ff4-22ed01570975\") "
Jul 01 17:01:46.478541 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.478330    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/96c32e70-bea8-41f0-8ff4-22ed01570975-metrics-client-ca\") pod \"96c32e70-bea8-41f0-8ff4-22ed01570975\" (UID: \"96c32e70-bea8-41f0-8ff4-22ed01570975\") "
Jul 01 17:01:46.478541 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.478395    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/96c32e70-bea8-41f0-8ff4-22ed01570975-config-volume\") pod \"96c32e70-bea8-41f0-8ff4-22ed01570975\" (UID: \"96c32e70-bea8-41f0-8ff4-22ed01570975\") "
Jul 01 17:01:46.478541 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.478432    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/96c32e70-bea8-41f0-8ff4-22ed01570975-config-out\") pod \"96c32e70-bea8-41f0-8ff4-22ed01570975\" (UID: \"96c32e70-bea8-41f0-8ff4-22ed01570975\") "
Jul 01 17:01:46.478541 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.478460    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/96c32e70-bea8-41f0-8ff4-22ed01570975-secret-alertmanager-main-tls\") pod \"96c32e70-bea8-41f0-8ff4-22ed01570975\" (UID: \"96c32e70-bea8-41f0-8ff4-22ed01570975\") "
Jul 01 17:01:46.478541 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.478498    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96c32e70-bea8-41f0-8ff4-22ed01570975-alertmanager-trusted-ca-bundle\") pod \"96c32e70-bea8-41f0-8ff4-22ed01570975\" (UID: \"96c32e70-bea8-41f0-8ff4-22ed01570975\") "
Jul 01 17:01:46.478880 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.478559    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/96c32e70-bea8-41f0-8ff4-22ed01570975-secret-alertmanager-kube-rbac-proxy\") pod \"96c32e70-bea8-41f0-8ff4-22ed01570975\" (UID: \"96c32e70-bea8-41f0-8ff4-22ed01570975\") "
Jul 01 17:01:46.478880 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.478586    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p8n8z\" (UniqueName: \"kubernetes.io/projected/96c32e70-bea8-41f0-8ff4-22ed01570975-kube-api-access-p8n8z\") pod \"96c32e70-bea8-41f0-8ff4-22ed01570975\" (UID: \"96c32e70-bea8-41f0-8ff4-22ed01570975\") "
Jul 01 17:01:46.478880 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.478619    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/96c32e70-bea8-41f0-8ff4-22ed01570975-web-config\") pod \"96c32e70-bea8-41f0-8ff4-22ed01570975\" (UID: \"96c32e70-bea8-41f0-8ff4-22ed01570975\") "
Jul 01 17:01:46.478880 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.478642    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/96c32e70-bea8-41f0-8ff4-22ed01570975-tls-assets\") pod \"96c32e70-bea8-41f0-8ff4-22ed01570975\" (UID: \"96c32e70-bea8-41f0-8ff4-22ed01570975\") "
Jul 01 17:01:46.478880 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.478675    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96c32e70-bea8-41f0-8ff4-22ed01570975-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "96c32e70-bea8-41f0-8ff4-22ed01570975" (UID: "96c32e70-bea8-41f0-8ff4-22ed01570975"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 17:01:46.478880 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.478815    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96c32e70-bea8-41f0-8ff4-22ed01570975-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "96c32e70-bea8-41f0-8ff4-22ed01570975" (UID: "96c32e70-bea8-41f0-8ff4-22ed01570975"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:01:46.478880 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.478877    2563 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/96c32e70-bea8-41f0-8ff4-22ed01570975-alertmanager-main-db\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:01:46.479419 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.479226    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96c32e70-bea8-41f0-8ff4-22ed01570975-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "96c32e70-bea8-41f0-8ff4-22ed01570975" (UID: "96c32e70-bea8-41f0-8ff4-22ed01570975"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:01:46.481625 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.481594    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96c32e70-bea8-41f0-8ff4-22ed01570975-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "96c32e70-bea8-41f0-8ff4-22ed01570975" (UID: "96c32e70-bea8-41f0-8ff4-22ed01570975"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 17:01:46.482241 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.482212    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96c32e70-bea8-41f0-8ff4-22ed01570975-kube-api-access-p8n8z" (OuterVolumeSpecName: "kube-api-access-p8n8z") pod "96c32e70-bea8-41f0-8ff4-22ed01570975" (UID: "96c32e70-bea8-41f0-8ff4-22ed01570975"). InnerVolumeSpecName "kube-api-access-p8n8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 17:01:46.482380 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.482264    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96c32e70-bea8-41f0-8ff4-22ed01570975-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "96c32e70-bea8-41f0-8ff4-22ed01570975" (UID: "96c32e70-bea8-41f0-8ff4-22ed01570975"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 17:01:46.482380 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.482333    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96c32e70-bea8-41f0-8ff4-22ed01570975-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "96c32e70-bea8-41f0-8ff4-22ed01570975" (UID: "96c32e70-bea8-41f0-8ff4-22ed01570975"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 17:01:46.482516 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.482377    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96c32e70-bea8-41f0-8ff4-22ed01570975-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "96c32e70-bea8-41f0-8ff4-22ed01570975" (UID: "96c32e70-bea8-41f0-8ff4-22ed01570975"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 17:01:46.482622 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.482596    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96c32e70-bea8-41f0-8ff4-22ed01570975-config-volume" (OuterVolumeSpecName: "config-volume") pod "96c32e70-bea8-41f0-8ff4-22ed01570975" (UID: "96c32e70-bea8-41f0-8ff4-22ed01570975"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 17:01:46.482717 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.482696    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96c32e70-bea8-41f0-8ff4-22ed01570975-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "96c32e70-bea8-41f0-8ff4-22ed01570975" (UID: "96c32e70-bea8-41f0-8ff4-22ed01570975"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 17:01:46.483049 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.483026    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96c32e70-bea8-41f0-8ff4-22ed01570975-config-out" (OuterVolumeSpecName: "config-out") pod "96c32e70-bea8-41f0-8ff4-22ed01570975" (UID: "96c32e70-bea8-41f0-8ff4-22ed01570975"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 17:01:46.485731 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.485641    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96c32e70-bea8-41f0-8ff4-22ed01570975-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "96c32e70-bea8-41f0-8ff4-22ed01570975" (UID: "96c32e70-bea8-41f0-8ff4-22ed01570975"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 17:01:46.492212 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.492189    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96c32e70-bea8-41f0-8ff4-22ed01570975-web-config" (OuterVolumeSpecName: "web-config") pod "96c32e70-bea8-41f0-8ff4-22ed01570975" (UID: "96c32e70-bea8-41f0-8ff4-22ed01570975"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 17:01:46.575027 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.574943    2563 generic.go:356] "Generic (PLEG): container finished" podID="96c32e70-bea8-41f0-8ff4-22ed01570975" containerID="139aa3f5026a5668904502bc4d7b383a21e6c19f8e79e75c18e6de5c8b36aad4" exitCode=0
Jul 01 17:01:46.575027 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.574968    2563 generic.go:356] "Generic (PLEG): container finished" podID="96c32e70-bea8-41f0-8ff4-22ed01570975" containerID="c60eaae264c31801ef956da594ebd41b7ae7a4c9873aa7f1e7ee689dca20fe14" exitCode=0
Jul 01 17:01:46.575027 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.575011    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"96c32e70-bea8-41f0-8ff4-22ed01570975","Type":"ContainerDied","Data":"139aa3f5026a5668904502bc4d7b383a21e6c19f8e79e75c18e6de5c8b36aad4"}
Jul 01 17:01:46.575027 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.575017    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:01:46.575245 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.575097    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"96c32e70-bea8-41f0-8ff4-22ed01570975","Type":"ContainerDied","Data":"c60eaae264c31801ef956da594ebd41b7ae7a4c9873aa7f1e7ee689dca20fe14"}
Jul 01 17:01:46.575245 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.575123    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"96c32e70-bea8-41f0-8ff4-22ed01570975","Type":"ContainerDied","Data":"152e15bdf685cd77e530650dbd134f3dc6fceaa3adee341ba70b3cce83805080"}
Jul 01 17:01:46.575245 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.575164    2563 scope.go:117] "RemoveContainer" containerID="6ff408cef50895fb958ff2adae8564ee0046d702987b0c0e12a6bf93f01641d8"
Jul 01 17:01:46.579313 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.579289    2563 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/96c32e70-bea8-41f0-8ff4-22ed01570975-config-volume\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:01:46.579313 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.579315    2563 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/96c32e70-bea8-41f0-8ff4-22ed01570975-config-out\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:01:46.579499 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.579328    2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/96c32e70-bea8-41f0-8ff4-22ed01570975-secret-alertmanager-main-tls\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:01:46.579499 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.579358    2563 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96c32e70-bea8-41f0-8ff4-22ed01570975-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:01:46.579499 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.579373    2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/96c32e70-bea8-41f0-8ff4-22ed01570975-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:01:46.579499 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.579386    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8n8z\" (UniqueName: \"kubernetes.io/projected/96c32e70-bea8-41f0-8ff4-22ed01570975-kube-api-access-p8n8z\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:01:46.579499 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.579425    2563 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/96c32e70-bea8-41f0-8ff4-22ed01570975-web-config\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:01:46.579499 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.579438    2563 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/96c32e70-bea8-41f0-8ff4-22ed01570975-tls-assets\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:01:46.579499 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.579460    2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/96c32e70-bea8-41f0-8ff4-22ed01570975-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:01:46.579499 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.579474    2563 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/96c32e70-bea8-41f0-8ff4-22ed01570975-cluster-tls-config\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:01:46.579499 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.579487    2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/96c32e70-bea8-41f0-8ff4-22ed01570975-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:01:46.579822 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.579509    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/96c32e70-bea8-41f0-8ff4-22ed01570975-metrics-client-ca\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:01:46.583200 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.583140    2563 scope.go:117] "RemoveContainer" containerID="139aa3f5026a5668904502bc4d7b383a21e6c19f8e79e75c18e6de5c8b36aad4"
Jul 01 17:01:46.590257 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.590241    2563 scope.go:117] "RemoveContainer" containerID="ea074ff1cca8524fdca9e75ca34f71764d603f1038ee02ca27db43f29ebee801"
Jul 01 17:01:46.597033 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.597010    2563 scope.go:117] "RemoveContainer" containerID="c60eaae264c31801ef956da594ebd41b7ae7a4c9873aa7f1e7ee689dca20fe14"
Jul 01 17:01:46.603924 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.603894    2563 scope.go:117] "RemoveContainer" containerID="b39ed92f5dcb65136c7765702081de61adbde5a51de65ad8ad3d913b0be7162a"
Jul 01 17:01:46.608451 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.608426    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 01 17:01:46.611615 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.611557    2563 scope.go:117] "RemoveContainer" containerID="33998e56bbbf155a14799fbb93eece630ad1dbb3f062c621ca12791f2c308f39"
Jul 01 17:01:46.613090 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.613070    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 01 17:01:46.620613 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.620595    2563 scope.go:117] "RemoveContainer" containerID="0fddaa616a1e3499b35dfc7d19a12aad5e52715c34caa2d7d050a1ea184ee7f3"
Jul 01 17:01:46.628317 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.628301    2563 scope.go:117] "RemoveContainer" containerID="6ff408cef50895fb958ff2adae8564ee0046d702987b0c0e12a6bf93f01641d8"
Jul 01 17:01:46.628624 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:01:46.628602    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ff408cef50895fb958ff2adae8564ee0046d702987b0c0e12a6bf93f01641d8\": container with ID starting with 6ff408cef50895fb958ff2adae8564ee0046d702987b0c0e12a6bf93f01641d8 not found: ID does not exist" containerID="6ff408cef50895fb958ff2adae8564ee0046d702987b0c0e12a6bf93f01641d8"
Jul 01 17:01:46.628673 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.628633    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ff408cef50895fb958ff2adae8564ee0046d702987b0c0e12a6bf93f01641d8"} err="failed to get container status \"6ff408cef50895fb958ff2adae8564ee0046d702987b0c0e12a6bf93f01641d8\": rpc error: code = NotFound desc = could not find container \"6ff408cef50895fb958ff2adae8564ee0046d702987b0c0e12a6bf93f01641d8\": container with ID starting with 6ff408cef50895fb958ff2adae8564ee0046d702987b0c0e12a6bf93f01641d8 not found: ID does not exist"
Jul 01 17:01:46.628673 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.628666    2563 scope.go:117] "RemoveContainer" containerID="139aa3f5026a5668904502bc4d7b383a21e6c19f8e79e75c18e6de5c8b36aad4"
Jul 01 17:01:46.628903 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:01:46.628884    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"139aa3f5026a5668904502bc4d7b383a21e6c19f8e79e75c18e6de5c8b36aad4\": container with ID starting with 139aa3f5026a5668904502bc4d7b383a21e6c19f8e79e75c18e6de5c8b36aad4 not found: ID does not exist" containerID="139aa3f5026a5668904502bc4d7b383a21e6c19f8e79e75c18e6de5c8b36aad4"
Jul 01 17:01:46.628957 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.628909    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"139aa3f5026a5668904502bc4d7b383a21e6c19f8e79e75c18e6de5c8b36aad4"} err="failed to get container status \"139aa3f5026a5668904502bc4d7b383a21e6c19f8e79e75c18e6de5c8b36aad4\": rpc error: code = NotFound desc = could not find container \"139aa3f5026a5668904502bc4d7b383a21e6c19f8e79e75c18e6de5c8b36aad4\": container with ID starting with 139aa3f5026a5668904502bc4d7b383a21e6c19f8e79e75c18e6de5c8b36aad4 not found: ID does not exist"
Jul 01 17:01:46.628957 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.628929    2563 scope.go:117] "RemoveContainer" containerID="ea074ff1cca8524fdca9e75ca34f71764d603f1038ee02ca27db43f29ebee801"
Jul 01 17:01:46.629147 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:01:46.629131    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea074ff1cca8524fdca9e75ca34f71764d603f1038ee02ca27db43f29ebee801\": container with ID starting with ea074ff1cca8524fdca9e75ca34f71764d603f1038ee02ca27db43f29ebee801 not found: ID does not exist" containerID="ea074ff1cca8524fdca9e75ca34f71764d603f1038ee02ca27db43f29ebee801"
Jul 01 17:01:46.629190 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.629151    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea074ff1cca8524fdca9e75ca34f71764d603f1038ee02ca27db43f29ebee801"} err="failed to get container status \"ea074ff1cca8524fdca9e75ca34f71764d603f1038ee02ca27db43f29ebee801\": rpc error: code = NotFound desc = could not find container \"ea074ff1cca8524fdca9e75ca34f71764d603f1038ee02ca27db43f29ebee801\": container with ID starting with ea074ff1cca8524fdca9e75ca34f71764d603f1038ee02ca27db43f29ebee801 not found: ID does not exist"
Jul 01 17:01:46.629190 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.629164    2563 scope.go:117] "RemoveContainer" containerID="c60eaae264c31801ef956da594ebd41b7ae7a4c9873aa7f1e7ee689dca20fe14"
Jul 01 17:01:46.629391 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:01:46.629375    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c60eaae264c31801ef956da594ebd41b7ae7a4c9873aa7f1e7ee689dca20fe14\": container with ID starting with c60eaae264c31801ef956da594ebd41b7ae7a4c9873aa7f1e7ee689dca20fe14 not found: ID does not exist" containerID="c60eaae264c31801ef956da594ebd41b7ae7a4c9873aa7f1e7ee689dca20fe14"
Jul 01 17:01:46.629437 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.629394    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c60eaae264c31801ef956da594ebd41b7ae7a4c9873aa7f1e7ee689dca20fe14"} err="failed to get container status \"c60eaae264c31801ef956da594ebd41b7ae7a4c9873aa7f1e7ee689dca20fe14\": rpc error: code = NotFound desc = could not find container \"c60eaae264c31801ef956da594ebd41b7ae7a4c9873aa7f1e7ee689dca20fe14\": container with ID starting with c60eaae264c31801ef956da594ebd41b7ae7a4c9873aa7f1e7ee689dca20fe14 not found: ID does not exist"
Jul 01 17:01:46.629437 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.629406    2563 scope.go:117] "RemoveContainer" containerID="b39ed92f5dcb65136c7765702081de61adbde5a51de65ad8ad3d913b0be7162a"
Jul 01 17:01:46.629645 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:01:46.629630    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b39ed92f5dcb65136c7765702081de61adbde5a51de65ad8ad3d913b0be7162a\": container with ID starting with b39ed92f5dcb65136c7765702081de61adbde5a51de65ad8ad3d913b0be7162a not found: ID does not exist" containerID="b39ed92f5dcb65136c7765702081de61adbde5a51de65ad8ad3d913b0be7162a"
Jul 01 17:01:46.629686 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.629650    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b39ed92f5dcb65136c7765702081de61adbde5a51de65ad8ad3d913b0be7162a"} err="failed to get container status \"b39ed92f5dcb65136c7765702081de61adbde5a51de65ad8ad3d913b0be7162a\": rpc error: code = NotFound desc = could not find container \"b39ed92f5dcb65136c7765702081de61adbde5a51de65ad8ad3d913b0be7162a\": container with ID starting with b39ed92f5dcb65136c7765702081de61adbde5a51de65ad8ad3d913b0be7162a not found: ID does not exist"
Jul 01 17:01:46.629686 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.629662    2563 scope.go:117] "RemoveContainer" containerID="33998e56bbbf155a14799fbb93eece630ad1dbb3f062c621ca12791f2c308f39"
Jul 01 17:01:46.629893 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:01:46.629876    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33998e56bbbf155a14799fbb93eece630ad1dbb3f062c621ca12791f2c308f39\": container with ID starting with 33998e56bbbf155a14799fbb93eece630ad1dbb3f062c621ca12791f2c308f39 not found: ID does not exist" containerID="33998e56bbbf155a14799fbb93eece630ad1dbb3f062c621ca12791f2c308f39"
Jul 01 17:01:46.629930 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.629897    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33998e56bbbf155a14799fbb93eece630ad1dbb3f062c621ca12791f2c308f39"} err="failed to get container status \"33998e56bbbf155a14799fbb93eece630ad1dbb3f062c621ca12791f2c308f39\": rpc error: code = NotFound desc = could not find container \"33998e56bbbf155a14799fbb93eece630ad1dbb3f062c621ca12791f2c308f39\": container with ID starting with 33998e56bbbf155a14799fbb93eece630ad1dbb3f062c621ca12791f2c308f39 not found: ID does not exist"
Jul 01 17:01:46.629930 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.629911    2563 scope.go:117] "RemoveContainer" containerID="0fddaa616a1e3499b35dfc7d19a12aad5e52715c34caa2d7d050a1ea184ee7f3"
Jul 01 17:01:46.630126 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:01:46.630111    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fddaa616a1e3499b35dfc7d19a12aad5e52715c34caa2d7d050a1ea184ee7f3\": container with ID starting with 0fddaa616a1e3499b35dfc7d19a12aad5e52715c34caa2d7d050a1ea184ee7f3 not found: ID does not exist" containerID="0fddaa616a1e3499b35dfc7d19a12aad5e52715c34caa2d7d050a1ea184ee7f3"
Jul 01 17:01:46.630164 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.630130    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fddaa616a1e3499b35dfc7d19a12aad5e52715c34caa2d7d050a1ea184ee7f3"} err="failed to get container status \"0fddaa616a1e3499b35dfc7d19a12aad5e52715c34caa2d7d050a1ea184ee7f3\": rpc error: code = NotFound desc = could not find container \"0fddaa616a1e3499b35dfc7d19a12aad5e52715c34caa2d7d050a1ea184ee7f3\": container with ID starting with 0fddaa616a1e3499b35dfc7d19a12aad5e52715c34caa2d7d050a1ea184ee7f3 not found: ID does not exist"
Jul 01 17:01:46.630164 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.630142    2563 scope.go:117] "RemoveContainer" containerID="6ff408cef50895fb958ff2adae8564ee0046d702987b0c0e12a6bf93f01641d8"
Jul 01 17:01:46.630376 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.630337    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ff408cef50895fb958ff2adae8564ee0046d702987b0c0e12a6bf93f01641d8"} err="failed to get container status \"6ff408cef50895fb958ff2adae8564ee0046d702987b0c0e12a6bf93f01641d8\": rpc error: code = NotFound desc = could not find container \"6ff408cef50895fb958ff2adae8564ee0046d702987b0c0e12a6bf93f01641d8\": container with ID starting with 6ff408cef50895fb958ff2adae8564ee0046d702987b0c0e12a6bf93f01641d8 not found: ID does not exist"
Jul 01 17:01:46.630433 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.630376    2563 scope.go:117] "RemoveContainer" containerID="139aa3f5026a5668904502bc4d7b383a21e6c19f8e79e75c18e6de5c8b36aad4"
Jul 01 17:01:46.630626 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.630608    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"139aa3f5026a5668904502bc4d7b383a21e6c19f8e79e75c18e6de5c8b36aad4"} err="failed to get container status \"139aa3f5026a5668904502bc4d7b383a21e6c19f8e79e75c18e6de5c8b36aad4\": rpc error: code = NotFound desc = could not find container \"139aa3f5026a5668904502bc4d7b383a21e6c19f8e79e75c18e6de5c8b36aad4\": container with ID starting with 139aa3f5026a5668904502bc4d7b383a21e6c19f8e79e75c18e6de5c8b36aad4 not found: ID does not exist"
Jul 01 17:01:46.630690 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.630627    2563 scope.go:117] "RemoveContainer" containerID="ea074ff1cca8524fdca9e75ca34f71764d603f1038ee02ca27db43f29ebee801"
Jul 01 17:01:46.630849 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.630832    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea074ff1cca8524fdca9e75ca34f71764d603f1038ee02ca27db43f29ebee801"} err="failed to get container status \"ea074ff1cca8524fdca9e75ca34f71764d603f1038ee02ca27db43f29ebee801\": rpc error: code = NotFound desc = could not find container \"ea074ff1cca8524fdca9e75ca34f71764d603f1038ee02ca27db43f29ebee801\": container with ID starting with ea074ff1cca8524fdca9e75ca34f71764d603f1038ee02ca27db43f29ebee801 not found: ID does not exist"
Jul 01 17:01:46.630890 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.630852    2563 scope.go:117] "RemoveContainer" containerID="c60eaae264c31801ef956da594ebd41b7ae7a4c9873aa7f1e7ee689dca20fe14"
Jul 01 17:01:46.631062 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.631047    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c60eaae264c31801ef956da594ebd41b7ae7a4c9873aa7f1e7ee689dca20fe14"} err="failed to get container status \"c60eaae264c31801ef956da594ebd41b7ae7a4c9873aa7f1e7ee689dca20fe14\": rpc error: code = NotFound desc = could not find container \"c60eaae264c31801ef956da594ebd41b7ae7a4c9873aa7f1e7ee689dca20fe14\": container with ID starting with c60eaae264c31801ef956da594ebd41b7ae7a4c9873aa7f1e7ee689dca20fe14 not found: ID does not exist"
Jul 01 17:01:46.631106 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.631063    2563 scope.go:117] "RemoveContainer" containerID="b39ed92f5dcb65136c7765702081de61adbde5a51de65ad8ad3d913b0be7162a"
Jul 01 17:01:46.631255 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.631238    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b39ed92f5dcb65136c7765702081de61adbde5a51de65ad8ad3d913b0be7162a"} err="failed to get container status \"b39ed92f5dcb65136c7765702081de61adbde5a51de65ad8ad3d913b0be7162a\": rpc error: code = NotFound desc = could not find container \"b39ed92f5dcb65136c7765702081de61adbde5a51de65ad8ad3d913b0be7162a\": container with ID starting with b39ed92f5dcb65136c7765702081de61adbde5a51de65ad8ad3d913b0be7162a not found: ID does not exist"
Jul 01 17:01:46.631323 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.631257    2563 scope.go:117] "RemoveContainer" containerID="33998e56bbbf155a14799fbb93eece630ad1dbb3f062c621ca12791f2c308f39"
Jul 01 17:01:46.631497 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.631478    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33998e56bbbf155a14799fbb93eece630ad1dbb3f062c621ca12791f2c308f39"} err="failed to get container status \"33998e56bbbf155a14799fbb93eece630ad1dbb3f062c621ca12791f2c308f39\": rpc error: code = NotFound desc = could not find container \"33998e56bbbf155a14799fbb93eece630ad1dbb3f062c621ca12791f2c308f39\": container with ID starting with 33998e56bbbf155a14799fbb93eece630ad1dbb3f062c621ca12791f2c308f39 not found: ID does not exist"
Jul 01 17:01:46.631546 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.631497    2563 scope.go:117] "RemoveContainer" containerID="0fddaa616a1e3499b35dfc7d19a12aad5e52715c34caa2d7d050a1ea184ee7f3"
Jul 01 17:01:46.631694 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.631676    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fddaa616a1e3499b35dfc7d19a12aad5e52715c34caa2d7d050a1ea184ee7f3"} err="failed to get container status \"0fddaa616a1e3499b35dfc7d19a12aad5e52715c34caa2d7d050a1ea184ee7f3\": rpc error: code = NotFound desc = could not find container \"0fddaa616a1e3499b35dfc7d19a12aad5e52715c34caa2d7d050a1ea184ee7f3\": container with ID starting with 0fddaa616a1e3499b35dfc7d19a12aad5e52715c34caa2d7d050a1ea184ee7f3 not found: ID does not exist"
Jul 01 17:01:46.643453 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.643366    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 01 17:01:46.643621 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.643610    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96c32e70-bea8-41f0-8ff4-22ed01570975" containerName="alertmanager"
Jul 01 17:01:46.643669 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.643623    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="96c32e70-bea8-41f0-8ff4-22ed01570975" containerName="alertmanager"
Jul 01 17:01:46.643669 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.643632    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96c32e70-bea8-41f0-8ff4-22ed01570975" containerName="kube-rbac-proxy-metric"
Jul 01 17:01:46.643669 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.643638    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="96c32e70-bea8-41f0-8ff4-22ed01570975" containerName="kube-rbac-proxy-metric"
Jul 01 17:01:46.643669 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.643644    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96c32e70-bea8-41f0-8ff4-22ed01570975" containerName="prom-label-proxy"
Jul 01 17:01:46.643669 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.643650    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="96c32e70-bea8-41f0-8ff4-22ed01570975" containerName="prom-label-proxy"
Jul 01 17:01:46.643669 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.643659    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96c32e70-bea8-41f0-8ff4-22ed01570975" containerName="kube-rbac-proxy"
Jul 01 17:01:46.643669 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.643664    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="96c32e70-bea8-41f0-8ff4-22ed01570975" containerName="kube-rbac-proxy"
Jul 01 17:01:46.643867 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.643676    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96c32e70-bea8-41f0-8ff4-22ed01570975" containerName="config-reloader"
Jul 01 17:01:46.643867 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.643681    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="96c32e70-bea8-41f0-8ff4-22ed01570975" containerName="config-reloader"
Jul 01 17:01:46.643867 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.643690    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96c32e70-bea8-41f0-8ff4-22ed01570975" containerName="init-config-reloader"
Jul 01 17:01:46.643867 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.643695    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="96c32e70-bea8-41f0-8ff4-22ed01570975" containerName="init-config-reloader"
Jul 01 17:01:46.643867 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.643700    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96c32e70-bea8-41f0-8ff4-22ed01570975" containerName="kube-rbac-proxy-web"
Jul 01 17:01:46.643867 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.643705    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="96c32e70-bea8-41f0-8ff4-22ed01570975" containerName="kube-rbac-proxy-web"
Jul 01 17:01:46.643867 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.643744    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="96c32e70-bea8-41f0-8ff4-22ed01570975" containerName="kube-rbac-proxy-web"
Jul 01 17:01:46.643867 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.643751    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="96c32e70-bea8-41f0-8ff4-22ed01570975" containerName="kube-rbac-proxy"
Jul 01 17:01:46.643867 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.643757    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="96c32e70-bea8-41f0-8ff4-22ed01570975" containerName="config-reloader"
Jul 01 17:01:46.643867 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.643763    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="96c32e70-bea8-41f0-8ff4-22ed01570975" containerName="kube-rbac-proxy-metric"
Jul 01 17:01:46.643867 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.643770    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="96c32e70-bea8-41f0-8ff4-22ed01570975" containerName="alertmanager"
Jul 01 17:01:46.643867 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.643774    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="96c32e70-bea8-41f0-8ff4-22ed01570975" containerName="prom-label-proxy"
Jul 01 17:01:46.648199 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.648181    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:01:46.652060 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.652025    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 01 17:01:46.652060 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.652058    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 01 17:01:46.652286 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.652025    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 01 17:01:46.652286 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.652269    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 01 17:01:46.652286 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.652272    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 01 17:01:46.652456 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.652431    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 01 17:01:46.652590 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.652572    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-n87xs\""
Jul 01 17:01:46.653422 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.653402    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 01 17:01:46.653530 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.653429    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 01 17:01:46.660020 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.660001    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 01 17:01:46.688583 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.688549    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 01 17:01:46.696272 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.696238    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96c32e70-bea8-41f0-8ff4-22ed01570975" path="/var/lib/kubelet/pods/96c32e70-bea8-41f0-8ff4-22ed01570975/volumes"
Jul 01 17:01:46.783584 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.783547    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/eb0a004c-c45c-4ea1-b6e0-6d26433107f0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"eb0a004c-c45c-4ea1-b6e0-6d26433107f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:01:46.783584 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.783586    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/eb0a004c-c45c-4ea1-b6e0-6d26433107f0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"eb0a004c-c45c-4ea1-b6e0-6d26433107f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:01:46.783796 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.783613    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/eb0a004c-c45c-4ea1-b6e0-6d26433107f0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"eb0a004c-c45c-4ea1-b6e0-6d26433107f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:01:46.783796 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.783637    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb0a004c-c45c-4ea1-b6e0-6d26433107f0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"eb0a004c-c45c-4ea1-b6e0-6d26433107f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:01:46.783796 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.783677    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/eb0a004c-c45c-4ea1-b6e0-6d26433107f0-config-volume\") pod \"alertmanager-main-0\" (UID: \"eb0a004c-c45c-4ea1-b6e0-6d26433107f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:01:46.783796 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.783699    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/eb0a004c-c45c-4ea1-b6e0-6d26433107f0-web-config\") pod \"alertmanager-main-0\" (UID: \"eb0a004c-c45c-4ea1-b6e0-6d26433107f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:01:46.783796 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.783735    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9gmq\" (UniqueName: \"kubernetes.io/projected/eb0a004c-c45c-4ea1-b6e0-6d26433107f0-kube-api-access-l9gmq\") pod \"alertmanager-main-0\" (UID: \"eb0a004c-c45c-4ea1-b6e0-6d26433107f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:01:46.783796 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.783755    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/eb0a004c-c45c-4ea1-b6e0-6d26433107f0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"eb0a004c-c45c-4ea1-b6e0-6d26433107f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:01:46.783796 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.783777    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/eb0a004c-c45c-4ea1-b6e0-6d26433107f0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"eb0a004c-c45c-4ea1-b6e0-6d26433107f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:01:46.784015 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.783798    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/eb0a004c-c45c-4ea1-b6e0-6d26433107f0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"eb0a004c-c45c-4ea1-b6e0-6d26433107f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:01:46.784015 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.783824    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/eb0a004c-c45c-4ea1-b6e0-6d26433107f0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"eb0a004c-c45c-4ea1-b6e0-6d26433107f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:01:46.784015 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.783878    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/eb0a004c-c45c-4ea1-b6e0-6d26433107f0-config-out\") pod \"alertmanager-main-0\" (UID: \"eb0a004c-c45c-4ea1-b6e0-6d26433107f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:01:46.784015 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.783904    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eb0a004c-c45c-4ea1-b6e0-6d26433107f0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"eb0a004c-c45c-4ea1-b6e0-6d26433107f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:01:46.884930 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.884831    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/eb0a004c-c45c-4ea1-b6e0-6d26433107f0-config-volume\") pod \"alertmanager-main-0\" (UID: \"eb0a004c-c45c-4ea1-b6e0-6d26433107f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:01:46.884930 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.884874    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/eb0a004c-c45c-4ea1-b6e0-6d26433107f0-web-config\") pod \"alertmanager-main-0\" (UID: \"eb0a004c-c45c-4ea1-b6e0-6d26433107f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:01:46.884930 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.884900    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9gmq\" (UniqueName: \"kubernetes.io/projected/eb0a004c-c45c-4ea1-b6e0-6d26433107f0-kube-api-access-l9gmq\") pod \"alertmanager-main-0\" (UID: \"eb0a004c-c45c-4ea1-b6e0-6d26433107f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:01:46.884930 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.884920    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/eb0a004c-c45c-4ea1-b6e0-6d26433107f0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"eb0a004c-c45c-4ea1-b6e0-6d26433107f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:01:46.885265 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.884951    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/eb0a004c-c45c-4ea1-b6e0-6d26433107f0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"eb0a004c-c45c-4ea1-b6e0-6d26433107f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:01:46.885265 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.885075    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/eb0a004c-c45c-4ea1-b6e0-6d26433107f0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"eb0a004c-c45c-4ea1-b6e0-6d26433107f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:01:46.885265 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.885116    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/eb0a004c-c45c-4ea1-b6e0-6d26433107f0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"eb0a004c-c45c-4ea1-b6e0-6d26433107f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:01:46.885265 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.885226    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/eb0a004c-c45c-4ea1-b6e0-6d26433107f0-config-out\") pod \"alertmanager-main-0\" (UID: \"eb0a004c-c45c-4ea1-b6e0-6d26433107f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:01:46.885497 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.885271    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eb0a004c-c45c-4ea1-b6e0-6d26433107f0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"eb0a004c-c45c-4ea1-b6e0-6d26433107f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:01:46.885497 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.885316    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/eb0a004c-c45c-4ea1-b6e0-6d26433107f0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"eb0a004c-c45c-4ea1-b6e0-6d26433107f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:01:46.885497 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.885376    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/eb0a004c-c45c-4ea1-b6e0-6d26433107f0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"eb0a004c-c45c-4ea1-b6e0-6d26433107f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:01:46.885497 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.885435    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/eb0a004c-c45c-4ea1-b6e0-6d26433107f0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"eb0a004c-c45c-4ea1-b6e0-6d26433107f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:01:46.885497 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.885461    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb0a004c-c45c-4ea1-b6e0-6d26433107f0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"eb0a004c-c45c-4ea1-b6e0-6d26433107f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:01:46.887394 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.886552    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb0a004c-c45c-4ea1-b6e0-6d26433107f0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"eb0a004c-c45c-4ea1-b6e0-6d26433107f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:01:46.887394 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.887127    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eb0a004c-c45c-4ea1-b6e0-6d26433107f0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"eb0a004c-c45c-4ea1-b6e0-6d26433107f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:01:46.888314 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.888205    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/eb0a004c-c45c-4ea1-b6e0-6d26433107f0-config-out\") pod \"alertmanager-main-0\" (UID: \"eb0a004c-c45c-4ea1-b6e0-6d26433107f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:01:46.888314 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.888209    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/eb0a004c-c45c-4ea1-b6e0-6d26433107f0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"eb0a004c-c45c-4ea1-b6e0-6d26433107f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:01:46.888314 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.888311    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/eb0a004c-c45c-4ea1-b6e0-6d26433107f0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"eb0a004c-c45c-4ea1-b6e0-6d26433107f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:01:46.888518 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.888405    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/eb0a004c-c45c-4ea1-b6e0-6d26433107f0-config-volume\") pod \"alertmanager-main-0\" (UID: \"eb0a004c-c45c-4ea1-b6e0-6d26433107f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:01:46.888518 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.888499    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/eb0a004c-c45c-4ea1-b6e0-6d26433107f0-web-config\") pod \"alertmanager-main-0\" (UID: \"eb0a004c-c45c-4ea1-b6e0-6d26433107f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:01:46.888598 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.888549    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/eb0a004c-c45c-4ea1-b6e0-6d26433107f0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"eb0a004c-c45c-4ea1-b6e0-6d26433107f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:01:46.888631 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.888612    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/eb0a004c-c45c-4ea1-b6e0-6d26433107f0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"eb0a004c-c45c-4ea1-b6e0-6d26433107f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:01:46.888664 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.888617    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/eb0a004c-c45c-4ea1-b6e0-6d26433107f0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"eb0a004c-c45c-4ea1-b6e0-6d26433107f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:01:46.889429 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.889412    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/eb0a004c-c45c-4ea1-b6e0-6d26433107f0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"eb0a004c-c45c-4ea1-b6e0-6d26433107f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:01:46.890387 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.890371    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/eb0a004c-c45c-4ea1-b6e0-6d26433107f0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"eb0a004c-c45c-4ea1-b6e0-6d26433107f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:01:46.894253 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.894233    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9gmq\" (UniqueName: \"kubernetes.io/projected/eb0a004c-c45c-4ea1-b6e0-6d26433107f0-kube-api-access-l9gmq\") pod \"alertmanager-main-0\" (UID: \"eb0a004c-c45c-4ea1-b6e0-6d26433107f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:01:46.957477 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:46.957447    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 01 17:01:47.101887 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:47.101862    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 01 17:01:47.103055 ip-10-0-137-180 kubenswrapper[2563]: W0701 17:01:47.103030    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb0a004c_c45c_4ea1_b6e0_6d26433107f0.slice/crio-fba0e817bbc09c33fd96e8bebfde9d3c8d9384007379f06d586eb527ad46722b WatchSource:0}: Error finding container fba0e817bbc09c33fd96e8bebfde9d3c8d9384007379f06d586eb527ad46722b: Status 404 returned error can't find the container with id fba0e817bbc09c33fd96e8bebfde9d3c8d9384007379f06d586eb527ad46722b
Jul 01 17:01:47.581416 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:47.581380    2563 generic.go:356] "Generic (PLEG): container finished" podID="eb0a004c-c45c-4ea1-b6e0-6d26433107f0" containerID="46d6c0c2796cfedae3719dc80693811a0d615d6cfbe9d6390f6a85fdefd33890" exitCode=0
Jul 01 17:01:47.581561 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:47.581469    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"eb0a004c-c45c-4ea1-b6e0-6d26433107f0","Type":"ContainerDied","Data":"46d6c0c2796cfedae3719dc80693811a0d615d6cfbe9d6390f6a85fdefd33890"}
Jul 01 17:01:47.581561 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:47.581543    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"eb0a004c-c45c-4ea1-b6e0-6d26433107f0","Type":"ContainerStarted","Data":"fba0e817bbc09c33fd96e8bebfde9d3c8d9384007379f06d586eb527ad46722b"}
Jul 01 17:01:48.588935 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:48.588894    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"eb0a004c-c45c-4ea1-b6e0-6d26433107f0","Type":"ContainerStarted","Data":"ef2bb22570f6f44183e9652e5fd95065b5a390bfbfc2c828212b7038cdcf9c16"}
Jul 01 17:01:48.589297 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:48.589010    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"eb0a004c-c45c-4ea1-b6e0-6d26433107f0","Type":"ContainerStarted","Data":"3a9430e95958ad0d7fa95defc8cd25b0ca61a70a3d8f4f595e982c14dd442f01"}
Jul 01 17:01:48.589297 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:48.589051    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"eb0a004c-c45c-4ea1-b6e0-6d26433107f0","Type":"ContainerStarted","Data":"b69b3bcda395901c252d02420bd13213b228a0682aa522ee8e9b191a0cbae730"}
Jul 01 17:01:48.589297 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:48.589089    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"eb0a004c-c45c-4ea1-b6e0-6d26433107f0","Type":"ContainerStarted","Data":"64772f32dae7392e84a2cad54950677df7fe1cd0333f1d0b3a714fab0086aea2"}
Jul 01 17:01:48.589297 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:48.589123    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"eb0a004c-c45c-4ea1-b6e0-6d26433107f0","Type":"ContainerStarted","Data":"9207bdcf071efe0736eed28bce156c33ff078159722aca1637e6e8c339e4baf9"}
Jul 01 17:01:48.589297 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:48.589156    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"eb0a004c-c45c-4ea1-b6e0-6d26433107f0","Type":"ContainerStarted","Data":"2649d1f6452ee566f7aa1917a32013b988daf1b3666a40033e6120a1f53632a7"}
Jul 01 17:01:48.619586 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:48.619536    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.6034878519999998 podStartE2EDuration="2.619519089s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.031237ms" isStatefulPod=false podCreationTimestamp="2026-07-01 17:01:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:01:48.618062332 +0000 UTC m=+226.415200769" watchObservedRunningTime="2026-07-01 17:01:48.619519089 +0000 UTC m=+226.416657521"
Jul 01 17:01:52.674975 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:52.674941    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:01:52.778030 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:52.777985    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:01:57.008780 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:57.008741    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:01:57.107906 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:57.107876    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:01:57.209835 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:57.209801    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:01:57.592156 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:57.592110    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:01:57.691552 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:01:57.691522    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:02:41.650061 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:02:41.650026    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:03:02.630194 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:03:02.630168    2563 kubelet.go:1648] "Image garbage collection succeeded"
Jul 01 17:03:07.813661 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:03:07.813630    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:03:07.907721 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:03:07.907682    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:03:08.131924 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:03:08.131834    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:03:08.230930 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:03:08.230893    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:03:08.331532 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:03:08.331490    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:03:36.024919 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:03:36.024879    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:03:36.164127 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:03:36.164091    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:03:38.254363 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:03:38.254320    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:03:38.401026 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:03:38.400989    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:03:42.028383 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:03:42.028333    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:03:42.206601 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:03:42.206540    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:03:42.208167 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:03:42.208144    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-68bff59c-4tgm7"]
Jul 01 17:03:42.215236 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:03:42.215210    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-68bff59c-4tgm7"
Jul 01 17:03:42.218064 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:03:42.218042    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 01 17:03:42.218182 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:03:42.218148    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 01 17:03:42.218823 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:03:42.218804    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 01 17:03:42.220167 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:03:42.220135    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Jul 01 17:03:42.220270 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:03:42.220151    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 01 17:03:42.220388 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:03:42.220376    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-s9zld\""
Jul 01 17:03:42.224547 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:03:42.224516    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-68bff59c-4tgm7"]
Jul 01 17:03:42.248379 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:03:42.248327    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/541a4f89-408b-4c38-a45d-af2eead85142-cabundle0\") pod \"keda-operator-68bff59c-4tgm7\" (UID: \"541a4f89-408b-4c38-a45d-af2eead85142\") " pod="openshift-keda/keda-operator-68bff59c-4tgm7"
Jul 01 17:03:42.248519 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:03:42.248401    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/541a4f89-408b-4c38-a45d-af2eead85142-certificates\") pod \"keda-operator-68bff59c-4tgm7\" (UID: \"541a4f89-408b-4c38-a45d-af2eead85142\") " pod="openshift-keda/keda-operator-68bff59c-4tgm7"
Jul 01 17:03:42.248519 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:03:42.248455    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbqjt\" (UniqueName: \"kubernetes.io/projected/541a4f89-408b-4c38-a45d-af2eead85142-kube-api-access-pbqjt\") pod \"keda-operator-68bff59c-4tgm7\" (UID: \"541a4f89-408b-4c38-a45d-af2eead85142\") " pod="openshift-keda/keda-operator-68bff59c-4tgm7"
Jul 01 17:03:42.312176 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:03:42.312091    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:03:42.349596 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:03:42.349559    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/541a4f89-408b-4c38-a45d-af2eead85142-certificates\") pod \"keda-operator-68bff59c-4tgm7\" (UID: \"541a4f89-408b-4c38-a45d-af2eead85142\") " pod="openshift-keda/keda-operator-68bff59c-4tgm7"
Jul 01 17:03:42.349799 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:03:42.349619    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbqjt\" (UniqueName: \"kubernetes.io/projected/541a4f89-408b-4c38-a45d-af2eead85142-kube-api-access-pbqjt\") pod \"keda-operator-68bff59c-4tgm7\" (UID: \"541a4f89-408b-4c38-a45d-af2eead85142\") " pod="openshift-keda/keda-operator-68bff59c-4tgm7"
Jul 01 17:03:42.349799 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:03:42.349643    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/541a4f89-408b-4c38-a45d-af2eead85142-cabundle0\") pod \"keda-operator-68bff59c-4tgm7\" (UID: \"541a4f89-408b-4c38-a45d-af2eead85142\") " pod="openshift-keda/keda-operator-68bff59c-4tgm7"
Jul 01 17:03:42.349799 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:03:42.349728    2563 secret.go:281] references non-existent secret key: ca.crt
Jul 01 17:03:42.349799 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:03:42.349751    2563 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 01 17:03:42.349799 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:03:42.349764    2563 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-68bff59c-4tgm7: references non-existent secret key: ca.crt
Jul 01 17:03:42.350051 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:03:42.349847    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/541a4f89-408b-4c38-a45d-af2eead85142-certificates podName:541a4f89-408b-4c38-a45d-af2eead85142 nodeName:}" failed. No retries permitted until 2026-07-01 17:03:42.849824444 +0000 UTC m=+340.646962836 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/541a4f89-408b-4c38-a45d-af2eead85142-certificates") pod "keda-operator-68bff59c-4tgm7" (UID: "541a4f89-408b-4c38-a45d-af2eead85142") : references non-existent secret key: ca.crt
Jul 01 17:03:42.350236 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:03:42.350218    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/541a4f89-408b-4c38-a45d-af2eead85142-cabundle0\") pod \"keda-operator-68bff59c-4tgm7\" (UID: \"541a4f89-408b-4c38-a45d-af2eead85142\") " pod="openshift-keda/keda-operator-68bff59c-4tgm7"
Jul 01 17:03:42.363268 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:03:42.363240    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbqjt\" (UniqueName: \"kubernetes.io/projected/541a4f89-408b-4c38-a45d-af2eead85142-kube-api-access-pbqjt\") pod \"keda-operator-68bff59c-4tgm7\" (UID: \"541a4f89-408b-4c38-a45d-af2eead85142\") " pod="openshift-keda/keda-operator-68bff59c-4tgm7"
Jul 01 17:03:42.429291 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:03:42.429254    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:03:42.461320 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:03:42.461280    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-559f75f947-5rzll"]
Jul 01 17:03:42.464668 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:03:42.464643    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-559f75f947-5rzll"
Jul 01 17:03:42.467520 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:03:42.467499    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Jul 01 17:03:42.475328 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:03:42.475303    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-559f75f947-5rzll"]
Jul 01 17:03:42.551633 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:03:42.551592    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/b532f374-d529-4174-aa52-b1f3e20e88e2-temp-vol\") pod \"keda-metrics-apiserver-559f75f947-5rzll\" (UID: \"b532f374-d529-4174-aa52-b1f3e20e88e2\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-5rzll"
Jul 01 17:03:42.551814 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:03:42.551660    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsz68\" (UniqueName: \"kubernetes.io/projected/b532f374-d529-4174-aa52-b1f3e20e88e2-kube-api-access-fsz68\") pod \"keda-metrics-apiserver-559f75f947-5rzll\" (UID: \"b532f374-d529-4174-aa52-b1f3e20e88e2\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-5rzll"
Jul 01 17:03:42.551814 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:03:42.551737    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/b532f374-d529-4174-aa52-b1f3e20e88e2-certificates\") pod \"keda-metrics-apiserver-559f75f947-5rzll\" (UID: \"b532f374-d529-4174-aa52-b1f3e20e88e2\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-5rzll"
Jul 01 17:03:42.653075 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:03:42.653035    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/b532f374-d529-4174-aa52-b1f3e20e88e2-certificates\") pod \"keda-metrics-apiserver-559f75f947-5rzll\" (UID: \"b532f374-d529-4174-aa52-b1f3e20e88e2\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-5rzll"
Jul 01 17:03:42.653075 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:03:42.653076    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/b532f374-d529-4174-aa52-b1f3e20e88e2-temp-vol\") pod \"keda-metrics-apiserver-559f75f947-5rzll\" (UID: \"b532f374-d529-4174-aa52-b1f3e20e88e2\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-5rzll"
Jul 01 17:03:42.653313 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:03:42.653112    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsz68\" (UniqueName: \"kubernetes.io/projected/b532f374-d529-4174-aa52-b1f3e20e88e2-kube-api-access-fsz68\") pod \"keda-metrics-apiserver-559f75f947-5rzll\" (UID: \"b532f374-d529-4174-aa52-b1f3e20e88e2\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-5rzll"
Jul 01 17:03:42.653313 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:03:42.653205    2563 secret.go:281] references non-existent secret key: tls.crt
Jul 01 17:03:42.653313 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:03:42.653230    2563 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Jul 01 17:03:42.653313 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:03:42.653248    2563 projected.go:266] Couldn't get secret openshift-keda/keda-metrics-apiserver-certs: secret "keda-metrics-apiserver-certs" not found
Jul 01 17:03:42.653313 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:03:42.653277    2563 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-559f75f947-5rzll: [references non-existent secret key: tls.crt, secret "keda-metrics-apiserver-certs" not found]
Jul 01 17:03:42.653526 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:03:42.653411    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b532f374-d529-4174-aa52-b1f3e20e88e2-certificates podName:b532f374-d529-4174-aa52-b1f3e20e88e2 nodeName:}" failed. No retries permitted until 2026-07-01 17:03:43.153388343 +0000 UTC m=+340.950526723 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/b532f374-d529-4174-aa52-b1f3e20e88e2-certificates") pod "keda-metrics-apiserver-559f75f947-5rzll" (UID: "b532f374-d529-4174-aa52-b1f3e20e88e2") : [references non-existent secret key: tls.crt, secret "keda-metrics-apiserver-certs" not found]
Jul 01 17:03:42.653568 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:03:42.653540    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/b532f374-d529-4174-aa52-b1f3e20e88e2-temp-vol\") pod \"keda-metrics-apiserver-559f75f947-5rzll\" (UID: \"b532f374-d529-4174-aa52-b1f3e20e88e2\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-5rzll"
Jul 01 17:03:42.664409 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:03:42.664378    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsz68\" (UniqueName: \"kubernetes.io/projected/b532f374-d529-4174-aa52-b1f3e20e88e2-kube-api-access-fsz68\") pod \"keda-metrics-apiserver-559f75f947-5rzll\" (UID: \"b532f374-d529-4174-aa52-b1f3e20e88e2\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-5rzll"
Jul 01 17:03:42.797881 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:03:42.797846    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:03:42.854431 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:03:42.854401    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/541a4f89-408b-4c38-a45d-af2eead85142-certificates\") pod \"keda-operator-68bff59c-4tgm7\" (UID: \"541a4f89-408b-4c38-a45d-af2eead85142\") " pod="openshift-keda/keda-operator-68bff59c-4tgm7"
Jul 01 17:03:42.857399 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:03:42.857372    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/541a4f89-408b-4c38-a45d-af2eead85142-certificates\") pod \"keda-operator-68bff59c-4tgm7\" (UID: \"541a4f89-408b-4c38-a45d-af2eead85142\") " pod="openshift-keda/keda-operator-68bff59c-4tgm7"
Jul 01 17:03:42.958241 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:03:42.958151    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:03:43.061228 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:03:43.061194    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:03:43.127954 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:03:43.127920    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-68bff59c-4tgm7"
Jul 01 17:03:43.157476 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:03:43.157447    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/b532f374-d529-4174-aa52-b1f3e20e88e2-certificates\") pod \"keda-metrics-apiserver-559f75f947-5rzll\" (UID: \"b532f374-d529-4174-aa52-b1f3e20e88e2\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-5rzll"
Jul 01 17:03:43.160309 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:03:43.160268    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/b532f374-d529-4174-aa52-b1f3e20e88e2-certificates\") pod \"keda-metrics-apiserver-559f75f947-5rzll\" (UID: \"b532f374-d529-4174-aa52-b1f3e20e88e2\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-5rzll"
Jul 01 17:03:43.294819 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:03:43.294788    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-68bff59c-4tgm7"]
Jul 01 17:03:43.298075 ip-10-0-137-180 kubenswrapper[2563]: W0701 17:03:43.298049    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod541a4f89_408b_4c38_a45d_af2eead85142.slice/crio-5083e507bab4b7101ac228fa738ad846ed08d689c607275f6d3265a454809e25 WatchSource:0}: Error finding container 5083e507bab4b7101ac228fa738ad846ed08d689c607275f6d3265a454809e25: Status 404 returned error can't find the container with id 5083e507bab4b7101ac228fa738ad846ed08d689c607275f6d3265a454809e25
Jul 01 17:03:43.299717 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:03:43.299697    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 01 17:03:43.375918 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:03:43.375891    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-559f75f947-5rzll"
Jul 01 17:03:43.437274 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:03:43.437248    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:03:43.517632 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:03:43.517610    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-559f75f947-5rzll"]
Jul 01 17:03:43.520151 ip-10-0-137-180 kubenswrapper[2563]: W0701 17:03:43.520116    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb532f374_d529_4174_aa52_b1f3e20e88e2.slice/crio-ca7da1bfabb73ed59841921085a196b6a111c5b048f4555d4b7a72bd137b1df7 WatchSource:0}: Error finding container ca7da1bfabb73ed59841921085a196b6a111c5b048f4555d4b7a72bd137b1df7: Status 404 returned error can't find the container with id ca7da1bfabb73ed59841921085a196b6a111c5b048f4555d4b7a72bd137b1df7
Jul 01 17:03:43.528117 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:03:43.528086    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:03:43.678711 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:03:43.678673    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:03:43.770988 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:03:43.770899    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:03:43.982421 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:03:43.982380    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-559f75f947-5rzll" event={"ID":"b532f374-d529-4174-aa52-b1f3e20e88e2","Type":"ContainerStarted","Data":"ca7da1bfabb73ed59841921085a196b6a111c5b048f4555d4b7a72bd137b1df7"}
Jul 01 17:03:43.983484 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:03:43.983461    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-68bff59c-4tgm7" event={"ID":"541a4f89-408b-4c38-a45d-af2eead85142","Type":"ContainerStarted","Data":"5083e507bab4b7101ac228fa738ad846ed08d689c607275f6d3265a454809e25"}
Jul 01 17:03:49.010636 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:03:49.010600    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-559f75f947-5rzll" event={"ID":"b532f374-d529-4174-aa52-b1f3e20e88e2","Type":"ContainerStarted","Data":"60d0085a3e8ae19078117822516fe632d313d8d6270abb770612ff04fe214e77"}
Jul 01 17:03:49.011030 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:03:49.010709    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-559f75f947-5rzll"
Jul 01 17:03:49.012151 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:03:49.012122    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-68bff59c-4tgm7" event={"ID":"541a4f89-408b-4c38-a45d-af2eead85142","Type":"ContainerStarted","Data":"445f0568b42d807f86e4b2e62c1057f182b491d3ba5347f68443137473db1c37"}
Jul 01 17:03:49.012266 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:03:49.012168    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-68bff59c-4tgm7"
Jul 01 17:03:49.029571 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:03:49.029527    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-559f75f947-5rzll" podStartSLOduration=2.472793355 podStartE2EDuration="7.029513316s" totalImagesPullingTime="4.556719961s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:03:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:03:49.028503016 +0000 UTC m=+346.825641430" watchObservedRunningTime="2026-07-01 17:03:49.029513316 +0000 UTC m=+346.826651717"
Jul 01 17:04:00.020862 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:04:00.020761    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-559f75f947-5rzll"
Jul 01 17:04:00.051581 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:04:00.051533    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-68bff59c-4tgm7" podStartSLOduration=13.268590119 podStartE2EDuration="18.051514998s" totalImagesPullingTime="4.782924879s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:03:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:03:49.044703792 +0000 UTC m=+346.841842217" watchObservedRunningTime="2026-07-01 17:04:00.051514998 +0000 UTC m=+357.848653399"
Jul 01 17:04:10.018394 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:04:10.018339    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-68bff59c-4tgm7"
Jul 01 17:05:03.000984 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:05:03.000947    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-84f4454459-kwltm"]
Jul 01 17:05:03.004030 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:05:03.004013    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-84f4454459-kwltm"
Jul 01 17:05:03.009663 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:05:03.009638    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 01 17:05:03.009800 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:05:03.009683    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-x72nx\""
Jul 01 17:05:03.010120 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:05:03.010107    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 01 17:05:03.011052 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:05:03.011038    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 01 17:05:03.019279 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:05:03.019258    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-84f4454459-kwltm"]
Jul 01 17:05:03.063072 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:05:03.063039    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/290532ea-031d-41b5-90e6-edf62584a6aa-cert\") pod \"kserve-controller-manager-84f4454459-kwltm\" (UID: \"290532ea-031d-41b5-90e6-edf62584a6aa\") " pod="kserve/kserve-controller-manager-84f4454459-kwltm"
Jul 01 17:05:03.063234 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:05:03.063115    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42nj7\" (UniqueName: \"kubernetes.io/projected/290532ea-031d-41b5-90e6-edf62584a6aa-kube-api-access-42nj7\") pod \"kserve-controller-manager-84f4454459-kwltm\" (UID: \"290532ea-031d-41b5-90e6-edf62584a6aa\") " pod="kserve/kserve-controller-manager-84f4454459-kwltm"
Jul 01 17:05:03.086157 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:05:03.086124    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:05:03.164171 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:05:03.164138    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/290532ea-031d-41b5-90e6-edf62584a6aa-cert\") pod \"kserve-controller-manager-84f4454459-kwltm\" (UID: \"290532ea-031d-41b5-90e6-edf62584a6aa\") " pod="kserve/kserve-controller-manager-84f4454459-kwltm"
Jul 01 17:05:03.164365 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:05:03.164188    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42nj7\" (UniqueName: \"kubernetes.io/projected/290532ea-031d-41b5-90e6-edf62584a6aa-kube-api-access-42nj7\") pod \"kserve-controller-manager-84f4454459-kwltm\" (UID: \"290532ea-031d-41b5-90e6-edf62584a6aa\") " pod="kserve/kserve-controller-manager-84f4454459-kwltm"
Jul 01 17:05:03.166672 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:05:03.166648    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/290532ea-031d-41b5-90e6-edf62584a6aa-cert\") pod \"kserve-controller-manager-84f4454459-kwltm\" (UID: \"290532ea-031d-41b5-90e6-edf62584a6aa\") " pod="kserve/kserve-controller-manager-84f4454459-kwltm"
Jul 01 17:05:03.176898 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:05:03.176872    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42nj7\" (UniqueName: \"kubernetes.io/projected/290532ea-031d-41b5-90e6-edf62584a6aa-kube-api-access-42nj7\") pod \"kserve-controller-manager-84f4454459-kwltm\" (UID: \"290532ea-031d-41b5-90e6-edf62584a6aa\") " pod="kserve/kserve-controller-manager-84f4454459-kwltm"
Jul 01 17:05:03.315113 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:05:03.315031    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-84f4454459-kwltm"
Jul 01 17:05:03.443997 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:05:03.443968    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-84f4454459-kwltm"]
Jul 01 17:05:03.447064 ip-10-0-137-180 kubenswrapper[2563]: W0701 17:05:03.447028    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod290532ea_031d_41b5_90e6_edf62584a6aa.slice/crio-6028961c4f188a3f1c46613a0cac67a372177483b91c621083afd3dd8fa3e2d6 WatchSource:0}: Error finding container 6028961c4f188a3f1c46613a0cac67a372177483b91c621083afd3dd8fa3e2d6: Status 404 returned error can't find the container with id 6028961c4f188a3f1c46613a0cac67a372177483b91c621083afd3dd8fa3e2d6
Jul 01 17:05:04.285516 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:05:04.285477    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-84f4454459-kwltm" event={"ID":"290532ea-031d-41b5-90e6-edf62584a6aa","Type":"ContainerStarted","Data":"6028961c4f188a3f1c46613a0cac67a372177483b91c621083afd3dd8fa3e2d6"}
Jul 01 17:05:05.238991 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:05:05.238930    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:05:06.294797 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:05:06.294763    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-84f4454459-kwltm" event={"ID":"290532ea-031d-41b5-90e6-edf62584a6aa","Type":"ContainerStarted","Data":"9609dc24892f7e8522a4ccdb08a8df82146c1f9523d1bbe6be2ea56fb1a8dd48"}
Jul 01 17:05:06.295193 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:05:06.294882    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-84f4454459-kwltm"
Jul 01 17:05:06.315285 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:05:06.315151    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-84f4454459-kwltm" podStartSLOduration=1.684141872 podStartE2EDuration="4.315132977s" totalImagesPullingTime="2.630991105s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:05:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:05:06.314850739 +0000 UTC m=+424.111989154" watchObservedRunningTime="2026-07-01 17:05:06.315132977 +0000 UTC m=+424.112271379"
Jul 01 17:05:19.205867 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:05:19.205828    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:05:24.754459 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:05:24.754367    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:05:24.841217 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:05:24.841179    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:05:37.304369 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:05:37.304316    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-84f4454459-kwltm"
Jul 01 17:05:42.034069 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:05:42.034037    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-84f4454459-kwltm"]
Jul 01 17:05:42.034500 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:05:42.034255    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve/kserve-controller-manager-84f4454459-kwltm" podUID="290532ea-031d-41b5-90e6-edf62584a6aa" containerName="manager" containerID="cri-o://9609dc24892f7e8522a4ccdb08a8df82146c1f9523d1bbe6be2ea56fb1a8dd48" gracePeriod=10
Jul 01 17:05:42.273661 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:05:42.273630    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-84f4454459-kwltm"
Jul 01 17:05:42.379014 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:05:42.378929    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/290532ea-031d-41b5-90e6-edf62584a6aa-cert\") pod \"290532ea-031d-41b5-90e6-edf62584a6aa\" (UID: \"290532ea-031d-41b5-90e6-edf62584a6aa\") "
Jul 01 17:05:42.379014 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:05:42.379009    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-42nj7\" (UniqueName: \"kubernetes.io/projected/290532ea-031d-41b5-90e6-edf62584a6aa-kube-api-access-42nj7\") pod \"290532ea-031d-41b5-90e6-edf62584a6aa\" (UID: \"290532ea-031d-41b5-90e6-edf62584a6aa\") "
Jul 01 17:05:42.381334 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:05:42.381305    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/290532ea-031d-41b5-90e6-edf62584a6aa-kube-api-access-42nj7" (OuterVolumeSpecName: "kube-api-access-42nj7") pod "290532ea-031d-41b5-90e6-edf62584a6aa" (UID: "290532ea-031d-41b5-90e6-edf62584a6aa"). InnerVolumeSpecName "kube-api-access-42nj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 17:05:42.381472 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:05:42.381305    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/290532ea-031d-41b5-90e6-edf62584a6aa-cert" (OuterVolumeSpecName: "cert") pod "290532ea-031d-41b5-90e6-edf62584a6aa" (UID: "290532ea-031d-41b5-90e6-edf62584a6aa"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 17:05:42.425200 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:05:42.425163    2563 generic.go:356] "Generic (PLEG): container finished" podID="290532ea-031d-41b5-90e6-edf62584a6aa" containerID="9609dc24892f7e8522a4ccdb08a8df82146c1f9523d1bbe6be2ea56fb1a8dd48" exitCode=0
Jul 01 17:05:42.425383 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:05:42.425210    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-84f4454459-kwltm"
Jul 01 17:05:42.425383 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:05:42.425239    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-84f4454459-kwltm" event={"ID":"290532ea-031d-41b5-90e6-edf62584a6aa","Type":"ContainerDied","Data":"9609dc24892f7e8522a4ccdb08a8df82146c1f9523d1bbe6be2ea56fb1a8dd48"}
Jul 01 17:05:42.425383 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:05:42.425291    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-84f4454459-kwltm" event={"ID":"290532ea-031d-41b5-90e6-edf62584a6aa","Type":"ContainerDied","Data":"6028961c4f188a3f1c46613a0cac67a372177483b91c621083afd3dd8fa3e2d6"}
Jul 01 17:05:42.425383 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:05:42.425320    2563 scope.go:117] "RemoveContainer" containerID="9609dc24892f7e8522a4ccdb08a8df82146c1f9523d1bbe6be2ea56fb1a8dd48"
Jul 01 17:05:42.433447 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:05:42.433429    2563 scope.go:117] "RemoveContainer" containerID="9609dc24892f7e8522a4ccdb08a8df82146c1f9523d1bbe6be2ea56fb1a8dd48"
Jul 01 17:05:42.433699 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:05:42.433679    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9609dc24892f7e8522a4ccdb08a8df82146c1f9523d1bbe6be2ea56fb1a8dd48\": container with ID starting with 9609dc24892f7e8522a4ccdb08a8df82146c1f9523d1bbe6be2ea56fb1a8dd48 not found: ID does not exist" containerID="9609dc24892f7e8522a4ccdb08a8df82146c1f9523d1bbe6be2ea56fb1a8dd48"
Jul 01 17:05:42.433741 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:05:42.433709    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9609dc24892f7e8522a4ccdb08a8df82146c1f9523d1bbe6be2ea56fb1a8dd48"} err="failed to get container status \"9609dc24892f7e8522a4ccdb08a8df82146c1f9523d1bbe6be2ea56fb1a8dd48\": rpc error: code = NotFound desc = could not find container \"9609dc24892f7e8522a4ccdb08a8df82146c1f9523d1bbe6be2ea56fb1a8dd48\": container with ID starting with 9609dc24892f7e8522a4ccdb08a8df82146c1f9523d1bbe6be2ea56fb1a8dd48 not found: ID does not exist"
Jul 01 17:05:42.446009 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:05:42.445987    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-84f4454459-kwltm"]
Jul 01 17:05:42.450992 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:05:42.450972    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-84f4454459-kwltm"]
Jul 01 17:05:42.480167 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:05:42.480137    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42nj7\" (UniqueName: \"kubernetes.io/projected/290532ea-031d-41b5-90e6-edf62584a6aa-kube-api-access-42nj7\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:05:42.480167 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:05:42.480162    2563 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/290532ea-031d-41b5-90e6-edf62584a6aa-cert\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:05:42.697084 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:05:42.697054    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="290532ea-031d-41b5-90e6-edf62584a6aa" path="/var/lib/kubelet/pods/290532ea-031d-41b5-90e6-edf62584a6aa/volumes"
Jul 01 17:05:43.155642 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:05:43.155609    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:05:43.233521 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:05:43.233485    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:05:46.719163 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:05:46.719125    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:05:47.356080 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:05:47.356037    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:05:47.420022 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:05:47.419989    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:05:54.024866 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:05:54.024823    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:05:54.145573 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:05:54.145537    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:05:54.237742 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:05:54.237707    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:06:17.752946 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:06:17.752899    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:06:17.800804 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:06:17.800728    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-fd65d7d6-swp72"]
Jul 01 17:06:17.801228 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:06:17.801213    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="290532ea-031d-41b5-90e6-edf62584a6aa" containerName="manager"
Jul 01 17:06:17.801271 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:06:17.801234    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="290532ea-031d-41b5-90e6-edf62584a6aa" containerName="manager"
Jul 01 17:06:17.801334 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:06:17.801323    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="290532ea-031d-41b5-90e6-edf62584a6aa" containerName="manager"
Jul 01 17:06:17.805578 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:06:17.805560    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-swp72"
Jul 01 17:06:17.808764 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:06:17.808732    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-84b54b7f97-b9kmn"]
Jul 01 17:06:17.809900 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:06:17.809877    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-phgcm\""
Jul 01 17:06:17.810295 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:06:17.810279    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 01 17:06:17.811050 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:06:17.811030    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 01 17:06:17.811148 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:06:17.811097    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 01 17:06:17.812001 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:06:17.811981    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-b9kmn"
Jul 01 17:06:17.814627 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:06:17.814607    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 01 17:06:17.815141 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:06:17.815122    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-w2gc5\""
Jul 01 17:06:17.820104 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:06:17.820081    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-swp72"]
Jul 01 17:06:17.825719 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:06:17.825553    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-b9kmn"]
Jul 01 17:06:17.848580 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:06:17.848545    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk98f\" (UniqueName: \"kubernetes.io/projected/612710ba-2417-4550-a5a8-d93667f657bf-kube-api-access-kk98f\") pod \"model-serving-api-fd65d7d6-swp72\" (UID: \"612710ba-2417-4550-a5a8-d93667f657bf\") " pod="kserve/model-serving-api-fd65d7d6-swp72"
Jul 01 17:06:17.848722 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:06:17.848597    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6s6v\" (UniqueName: \"kubernetes.io/projected/7062d5e0-9b0b-4338-bb97-1a79a522f823-kube-api-access-g6s6v\") pod \"odh-model-controller-84b54b7f97-b9kmn\" (UID: \"7062d5e0-9b0b-4338-bb97-1a79a522f823\") " pod="kserve/odh-model-controller-84b54b7f97-b9kmn"
Jul 01 17:06:17.848722 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:06:17.848653    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7062d5e0-9b0b-4338-bb97-1a79a522f823-cert\") pod \"odh-model-controller-84b54b7f97-b9kmn\" (UID: \"7062d5e0-9b0b-4338-bb97-1a79a522f823\") " pod="kserve/odh-model-controller-84b54b7f97-b9kmn"
Jul 01 17:06:17.848722 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:06:17.848702    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/612710ba-2417-4550-a5a8-d93667f657bf-tls-certs\") pod \"model-serving-api-fd65d7d6-swp72\" (UID: \"612710ba-2417-4550-a5a8-d93667f657bf\") " pod="kserve/model-serving-api-fd65d7d6-swp72"
Jul 01 17:06:17.854901 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:06:17.854868    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:06:17.949632 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:06:17.949588    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/612710ba-2417-4550-a5a8-d93667f657bf-tls-certs\") pod \"model-serving-api-fd65d7d6-swp72\" (UID: \"612710ba-2417-4550-a5a8-d93667f657bf\") " pod="kserve/model-serving-api-fd65d7d6-swp72"
Jul 01 17:06:17.949836 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:06:17.949644    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk98f\" (UniqueName: \"kubernetes.io/projected/612710ba-2417-4550-a5a8-d93667f657bf-kube-api-access-kk98f\") pod \"model-serving-api-fd65d7d6-swp72\" (UID: \"612710ba-2417-4550-a5a8-d93667f657bf\") " pod="kserve/model-serving-api-fd65d7d6-swp72"
Jul 01 17:06:17.949836 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:06:17.949687    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6s6v\" (UniqueName: \"kubernetes.io/projected/7062d5e0-9b0b-4338-bb97-1a79a522f823-kube-api-access-g6s6v\") pod \"odh-model-controller-84b54b7f97-b9kmn\" (UID: \"7062d5e0-9b0b-4338-bb97-1a79a522f823\") " pod="kserve/odh-model-controller-84b54b7f97-b9kmn"
Jul 01 17:06:17.949836 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:06:17.949708    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7062d5e0-9b0b-4338-bb97-1a79a522f823-cert\") pod \"odh-model-controller-84b54b7f97-b9kmn\" (UID: \"7062d5e0-9b0b-4338-bb97-1a79a522f823\") " pod="kserve/odh-model-controller-84b54b7f97-b9kmn"
Jul 01 17:06:17.949836 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:06:17.949748    2563 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
Jul 01 17:06:17.950043 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:06:17.949844    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/612710ba-2417-4550-a5a8-d93667f657bf-tls-certs podName:612710ba-2417-4550-a5a8-d93667f657bf nodeName:}" failed. No retries permitted until 2026-07-01 17:06:18.449823691 +0000 UTC m=+496.246962070 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/612710ba-2417-4550-a5a8-d93667f657bf-tls-certs") pod "model-serving-api-fd65d7d6-swp72" (UID: "612710ba-2417-4550-a5a8-d93667f657bf") : secret "model-serving-api-tls" not found
Jul 01 17:06:17.952282 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:06:17.952251    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7062d5e0-9b0b-4338-bb97-1a79a522f823-cert\") pod \"odh-model-controller-84b54b7f97-b9kmn\" (UID: \"7062d5e0-9b0b-4338-bb97-1a79a522f823\") " pod="kserve/odh-model-controller-84b54b7f97-b9kmn"
Jul 01 17:06:17.970394 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:06:17.970367    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6s6v\" (UniqueName: \"kubernetes.io/projected/7062d5e0-9b0b-4338-bb97-1a79a522f823-kube-api-access-g6s6v\") pod \"odh-model-controller-84b54b7f97-b9kmn\" (UID: \"7062d5e0-9b0b-4338-bb97-1a79a522f823\") " pod="kserve/odh-model-controller-84b54b7f97-b9kmn"
Jul 01 17:06:17.983542 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:06:17.983505    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk98f\" (UniqueName: \"kubernetes.io/projected/612710ba-2417-4550-a5a8-d93667f657bf-kube-api-access-kk98f\") pod \"model-serving-api-fd65d7d6-swp72\" (UID: \"612710ba-2417-4550-a5a8-d93667f657bf\") " pod="kserve/model-serving-api-fd65d7d6-swp72"
Jul 01 17:06:18.030724 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:06:18.030629    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:06:18.127183 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:06:18.127150    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-b9kmn"
Jul 01 17:06:18.230823 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:06:18.230765    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:06:18.259889 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:06:18.259852    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-b9kmn"]
Jul 01 17:06:18.263952 ip-10-0-137-180 kubenswrapper[2563]: W0701 17:06:18.263913    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7062d5e0_9b0b_4338_bb97_1a79a522f823.slice/crio-4a51b4c7006c9543f4778540310b086f89bb8853434e23510afe472ebd40a324 WatchSource:0}: Error finding container 4a51b4c7006c9543f4778540310b086f89bb8853434e23510afe472ebd40a324: Status 404 returned error can't find the container with id 4a51b4c7006c9543f4778540310b086f89bb8853434e23510afe472ebd40a324
Jul 01 17:06:18.453727 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:06:18.453647    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/612710ba-2417-4550-a5a8-d93667f657bf-tls-certs\") pod \"model-serving-api-fd65d7d6-swp72\" (UID: \"612710ba-2417-4550-a5a8-d93667f657bf\") " pod="kserve/model-serving-api-fd65d7d6-swp72"
Jul 01 17:06:18.456084 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:06:18.456065    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/612710ba-2417-4550-a5a8-d93667f657bf-tls-certs\") pod \"model-serving-api-fd65d7d6-swp72\" (UID: \"612710ba-2417-4550-a5a8-d93667f657bf\") " pod="kserve/model-serving-api-fd65d7d6-swp72"
Jul 01 17:06:18.559395 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:06:18.559357    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-b9kmn" event={"ID":"7062d5e0-9b0b-4338-bb97-1a79a522f823","Type":"ContainerStarted","Data":"4a51b4c7006c9543f4778540310b086f89bb8853434e23510afe472ebd40a324"}
Jul 01 17:06:18.601006 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:06:18.600958    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:06:18.684195 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:06:18.684159    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:06:18.717138 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:06:18.717110    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-swp72"
Jul 01 17:06:18.850512 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:06:18.850369    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-swp72"]
Jul 01 17:06:18.852710 ip-10-0-137-180 kubenswrapper[2563]: W0701 17:06:18.852684    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod612710ba_2417_4550_a5a8_d93667f657bf.slice/crio-3679ca195eea3f74068b9fdfe2167ab5a5d3487cb75e88557f9a30b9f3958fa3 WatchSource:0}: Error finding container 3679ca195eea3f74068b9fdfe2167ab5a5d3487cb75e88557f9a30b9f3958fa3: Status 404 returned error can't find the container with id 3679ca195eea3f74068b9fdfe2167ab5a5d3487cb75e88557f9a30b9f3958fa3
Jul 01 17:06:19.568245 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:06:19.568150    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-swp72" event={"ID":"612710ba-2417-4550-a5a8-d93667f657bf","Type":"ContainerStarted","Data":"3679ca195eea3f74068b9fdfe2167ab5a5d3487cb75e88557f9a30b9f3958fa3"}
Jul 01 17:06:22.583653 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:06:22.583614    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-b9kmn" event={"ID":"7062d5e0-9b0b-4338-bb97-1a79a522f823","Type":"ContainerStarted","Data":"359d64f8eeba8ec458b297a3cc52303fa55034c9e3a181159d928eaa34cf0421"}
Jul 01 17:06:22.584146 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:06:22.583682    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-84b54b7f97-b9kmn"
Jul 01 17:06:22.585123 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:06:22.585103    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-swp72" event={"ID":"612710ba-2417-4550-a5a8-d93667f657bf","Type":"ContainerStarted","Data":"ca876c2a6bd802f2b44d0b356845002221d054abdd9c6849793dd073cb205779"}
Jul 01 17:06:22.585237 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:06:22.585225    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-fd65d7d6-swp72"
Jul 01 17:06:22.609070 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:06:22.609015    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-84b54b7f97-b9kmn" podStartSLOduration=1.700821839 podStartE2EDuration="5.609000929s" totalImagesPullingTime="3.90817909s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:06:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:06:22.607569015 +0000 UTC m=+500.404707441" watchObservedRunningTime="2026-07-01 17:06:22.609000929 +0000 UTC m=+500.406139327"
Jul 01 17:06:22.631726 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:06:22.631670    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-fd65d7d6-swp72" podStartSLOduration=2.261678895 podStartE2EDuration="5.631651657s" totalImagesPullingTime="3.369972762s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:06:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:06:22.630425074 +0000 UTC m=+500.427563494" watchObservedRunningTime="2026-07-01 17:06:22.631651657 +0000 UTC m=+500.428790065"
Jul 01 17:06:33.591590 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:06:33.591558    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-84b54b7f97-b9kmn"
Jul 01 17:06:33.593567 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:06:33.593546    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-fd65d7d6-swp72"
Jul 01 17:07:54.016981 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:07:54.016944    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:07:54.106967 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:07:54.106934    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:08:06.130920 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:08:06.130879    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:08:06.376981 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:08:06.376935    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:08:06.399880 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:08:06.399803    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1b15a-predictor-84b9dd6865-v2nnf"]
Jul 01 17:08:06.403061 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:08:06.403042    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-1b15a-predictor-84b9dd6865-v2nnf"
Jul 01 17:08:06.405874 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:08:06.405846    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-1b15a-predictor-serving-cert\""
Jul 01 17:08:06.405994 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:08:06.405882    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 01 17:08:06.405994 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:08:06.405933    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-csvp5\""
Jul 01 17:08:06.406958 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:08:06.406935    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-1b15a-kube-rbac-proxy-sar-config\""
Jul 01 17:08:06.407053 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:08:06.406946    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 01 17:08:06.420470 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:08:06.420449    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1b15a-predictor-84b9dd6865-v2nnf"]
Jul 01 17:08:06.511783 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:08:06.511748    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:08:06.535697 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:08:06.535667    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/440dce6a-9f80-46d4-a092-9a34c40759cf-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-1b15a-predictor-84b9dd6865-v2nnf\" (UID: \"440dce6a-9f80-46d4-a092-9a34c40759cf\") " pod="kserve-ci-e2e-test/success-200-isvc-1b15a-predictor-84b9dd6865-v2nnf"
Jul 01 17:08:06.535855 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:08:06.535723    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49h79\" (UniqueName: \"kubernetes.io/projected/440dce6a-9f80-46d4-a092-9a34c40759cf-kube-api-access-49h79\") pod \"success-200-isvc-1b15a-predictor-84b9dd6865-v2nnf\" (UID: \"440dce6a-9f80-46d4-a092-9a34c40759cf\") " pod="kserve-ci-e2e-test/success-200-isvc-1b15a-predictor-84b9dd6865-v2nnf"
Jul 01 17:08:06.535855 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:08:06.535755    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/440dce6a-9f80-46d4-a092-9a34c40759cf-proxy-tls\") pod \"success-200-isvc-1b15a-predictor-84b9dd6865-v2nnf\" (UID: \"440dce6a-9f80-46d4-a092-9a34c40759cf\") " pod="kserve-ci-e2e-test/success-200-isvc-1b15a-predictor-84b9dd6865-v2nnf"
Jul 01 17:08:06.626282 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:08:06.626249    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:08:06.636238 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:08:06.636211    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/440dce6a-9f80-46d4-a092-9a34c40759cf-proxy-tls\") pod \"success-200-isvc-1b15a-predictor-84b9dd6865-v2nnf\" (UID: \"440dce6a-9f80-46d4-a092-9a34c40759cf\") " pod="kserve-ci-e2e-test/success-200-isvc-1b15a-predictor-84b9dd6865-v2nnf"
Jul 01 17:08:06.636407 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:08:06.636262    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/440dce6a-9f80-46d4-a092-9a34c40759cf-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-1b15a-predictor-84b9dd6865-v2nnf\" (UID: \"440dce6a-9f80-46d4-a092-9a34c40759cf\") " pod="kserve-ci-e2e-test/success-200-isvc-1b15a-predictor-84b9dd6865-v2nnf"
Jul 01 17:08:06.636479 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:08:06.636397    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49h79\" (UniqueName: \"kubernetes.io/projected/440dce6a-9f80-46d4-a092-9a34c40759cf-kube-api-access-49h79\") pod \"success-200-isvc-1b15a-predictor-84b9dd6865-v2nnf\" (UID: \"440dce6a-9f80-46d4-a092-9a34c40759cf\") " pod="kserve-ci-e2e-test/success-200-isvc-1b15a-predictor-84b9dd6865-v2nnf"
Jul 01 17:08:06.637069 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:08:06.637051    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/440dce6a-9f80-46d4-a092-9a34c40759cf-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-1b15a-predictor-84b9dd6865-v2nnf\" (UID: \"440dce6a-9f80-46d4-a092-9a34c40759cf\") " pod="kserve-ci-e2e-test/success-200-isvc-1b15a-predictor-84b9dd6865-v2nnf"
Jul 01 17:08:06.639186 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:08:06.639167    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/440dce6a-9f80-46d4-a092-9a34c40759cf-proxy-tls\") pod \"success-200-isvc-1b15a-predictor-84b9dd6865-v2nnf\" (UID: \"440dce6a-9f80-46d4-a092-9a34c40759cf\") " pod="kserve-ci-e2e-test/success-200-isvc-1b15a-predictor-84b9dd6865-v2nnf"
Jul 01 17:08:06.654629 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:08:06.654572    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49h79\" (UniqueName: \"kubernetes.io/projected/440dce6a-9f80-46d4-a092-9a34c40759cf-kube-api-access-49h79\") pod \"success-200-isvc-1b15a-predictor-84b9dd6865-v2nnf\" (UID: \"440dce6a-9f80-46d4-a092-9a34c40759cf\") " pod="kserve-ci-e2e-test/success-200-isvc-1b15a-predictor-84b9dd6865-v2nnf"
Jul 01 17:08:06.702336 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:08:06.702310    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1b15a-predictor-5c76b7cdd8-56vrx"]
Jul 01 17:08:06.705612 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:08:06.705597    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1b15a-predictor-5c76b7cdd8-56vrx"
Jul 01 17:08:06.709234 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:08:06.709206    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-1b15a-kube-rbac-proxy-sar-config\""
Jul 01 17:08:06.709453 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:08:06.709209    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-1b15a-predictor-serving-cert\""
Jul 01 17:08:06.714391 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:08:06.714373    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-1b15a-predictor-84b9dd6865-v2nnf"
Jul 01 17:08:06.723387 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:08:06.723365    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1b15a-predictor-5c76b7cdd8-56vrx"]
Jul 01 17:08:06.749483 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:08:06.749454    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:08:06.839402 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:08:06.838103    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmctr\" (UniqueName: \"kubernetes.io/projected/22a5235c-0a90-4a58-b636-3c53080e34c7-kube-api-access-mmctr\") pod \"error-404-isvc-1b15a-predictor-5c76b7cdd8-56vrx\" (UID: \"22a5235c-0a90-4a58-b636-3c53080e34c7\") " pod="kserve-ci-e2e-test/error-404-isvc-1b15a-predictor-5c76b7cdd8-56vrx"
Jul 01 17:08:06.839402 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:08:06.838154    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22a5235c-0a90-4a58-b636-3c53080e34c7-proxy-tls\") pod \"error-404-isvc-1b15a-predictor-5c76b7cdd8-56vrx\" (UID: \"22a5235c-0a90-4a58-b636-3c53080e34c7\") " pod="kserve-ci-e2e-test/error-404-isvc-1b15a-predictor-5c76b7cdd8-56vrx"
Jul 01 17:08:06.839402 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:08:06.838226    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22a5235c-0a90-4a58-b636-3c53080e34c7-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-1b15a-predictor-5c76b7cdd8-56vrx\" (UID: \"22a5235c-0a90-4a58-b636-3c53080e34c7\") " pod="kserve-ci-e2e-test/error-404-isvc-1b15a-predictor-5c76b7cdd8-56vrx"
Jul 01 17:08:06.845509 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:08:06.845464    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:08:06.876498 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:08:06.876472    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1b15a-predictor-84b9dd6865-v2nnf"]
Jul 01 17:08:06.879467 ip-10-0-137-180 kubenswrapper[2563]: W0701 17:08:06.879434    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod440dce6a_9f80_46d4_a092_9a34c40759cf.slice/crio-7119fea76ce34a57fbf6cb53ea097c66e20cc1193cccf4cca6c44732820dac7e WatchSource:0}: Error finding container 7119fea76ce34a57fbf6cb53ea097c66e20cc1193cccf4cca6c44732820dac7e: Status 404 returned error can't find the container with id 7119fea76ce34a57fbf6cb53ea097c66e20cc1193cccf4cca6c44732820dac7e
Jul 01 17:08:06.939125 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:08:06.939034    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmctr\" (UniqueName: \"kubernetes.io/projected/22a5235c-0a90-4a58-b636-3c53080e34c7-kube-api-access-mmctr\") pod \"error-404-isvc-1b15a-predictor-5c76b7cdd8-56vrx\" (UID: \"22a5235c-0a90-4a58-b636-3c53080e34c7\") " pod="kserve-ci-e2e-test/error-404-isvc-1b15a-predictor-5c76b7cdd8-56vrx"
Jul 01 17:08:06.939125 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:08:06.939072    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22a5235c-0a90-4a58-b636-3c53080e34c7-proxy-tls\") pod \"error-404-isvc-1b15a-predictor-5c76b7cdd8-56vrx\" (UID: \"22a5235c-0a90-4a58-b636-3c53080e34c7\") " pod="kserve-ci-e2e-test/error-404-isvc-1b15a-predictor-5c76b7cdd8-56vrx"
Jul 01 17:08:06.939125 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:08:06.939096    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22a5235c-0a90-4a58-b636-3c53080e34c7-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-1b15a-predictor-5c76b7cdd8-56vrx\" (UID: \"22a5235c-0a90-4a58-b636-3c53080e34c7\") " pod="kserve-ci-e2e-test/error-404-isvc-1b15a-predictor-5c76b7cdd8-56vrx"
Jul 01 17:08:06.939442 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:08:06.939227    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-1b15a-predictor-serving-cert: secret "error-404-isvc-1b15a-predictor-serving-cert" not found
Jul 01 17:08:06.939442 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:08:06.939311    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22a5235c-0a90-4a58-b636-3c53080e34c7-proxy-tls podName:22a5235c-0a90-4a58-b636-3c53080e34c7 nodeName:}" failed. No retries permitted until 2026-07-01 17:08:07.439286654 +0000 UTC m=+605.236425039 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/22a5235c-0a90-4a58-b636-3c53080e34c7-proxy-tls") pod "error-404-isvc-1b15a-predictor-5c76b7cdd8-56vrx" (UID: "22a5235c-0a90-4a58-b636-3c53080e34c7") : secret "error-404-isvc-1b15a-predictor-serving-cert" not found
Jul 01 17:08:06.939779 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:08:06.939755    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22a5235c-0a90-4a58-b636-3c53080e34c7-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-1b15a-predictor-5c76b7cdd8-56vrx\" (UID: \"22a5235c-0a90-4a58-b636-3c53080e34c7\") " pod="kserve-ci-e2e-test/error-404-isvc-1b15a-predictor-5c76b7cdd8-56vrx"
Jul 01 17:08:06.949913 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:08:06.949872    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmctr\" (UniqueName: \"kubernetes.io/projected/22a5235c-0a90-4a58-b636-3c53080e34c7-kube-api-access-mmctr\") pod \"error-404-isvc-1b15a-predictor-5c76b7cdd8-56vrx\" (UID: \"22a5235c-0a90-4a58-b636-3c53080e34c7\") " pod="kserve-ci-e2e-test/error-404-isvc-1b15a-predictor-5c76b7cdd8-56vrx"
Jul 01 17:08:06.962188 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:08:06.962157    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:08:06.973456 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:08:06.973422    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1b15a-predictor-84b9dd6865-v2nnf" event={"ID":"440dce6a-9f80-46d4-a092-9a34c40759cf","Type":"ContainerStarted","Data":"7119fea76ce34a57fbf6cb53ea097c66e20cc1193cccf4cca6c44732820dac7e"}
Jul 01 17:08:07.304881 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:08:07.304840    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:08:07.444044 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:08:07.444008    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22a5235c-0a90-4a58-b636-3c53080e34c7-proxy-tls\") pod \"error-404-isvc-1b15a-predictor-5c76b7cdd8-56vrx\" (UID: \"22a5235c-0a90-4a58-b636-3c53080e34c7\") " pod="kserve-ci-e2e-test/error-404-isvc-1b15a-predictor-5c76b7cdd8-56vrx"
Jul 01 17:08:07.446543 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:08:07.446524    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22a5235c-0a90-4a58-b636-3c53080e34c7-proxy-tls\") pod \"error-404-isvc-1b15a-predictor-5c76b7cdd8-56vrx\" (UID: \"22a5235c-0a90-4a58-b636-3c53080e34c7\") " pod="kserve-ci-e2e-test/error-404-isvc-1b15a-predictor-5c76b7cdd8-56vrx"
Jul 01 17:08:07.448866 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:08:07.448837    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:08:07.617001 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:08:07.616916    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1b15a-predictor-5c76b7cdd8-56vrx"
Jul 01 17:08:07.763181 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:08:07.763156    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1b15a-predictor-5c76b7cdd8-56vrx"]
Jul 01 17:08:07.764851 ip-10-0-137-180 kubenswrapper[2563]: W0701 17:08:07.764825    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22a5235c_0a90_4a58_b636_3c53080e34c7.slice/crio-90394b9b49ee3904fda592277edeee7005ffaa7d35b158ad33939b662843b6e3 WatchSource:0}: Error finding container 90394b9b49ee3904fda592277edeee7005ffaa7d35b158ad33939b662843b6e3: Status 404 returned error can't find the container with id 90394b9b49ee3904fda592277edeee7005ffaa7d35b158ad33939b662843b6e3
Jul 01 17:08:07.978362 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:08:07.978317    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1b15a-predictor-5c76b7cdd8-56vrx" event={"ID":"22a5235c-0a90-4a58-b636-3c53080e34c7","Type":"ContainerStarted","Data":"90394b9b49ee3904fda592277edeee7005ffaa7d35b158ad33939b662843b6e3"}
Jul 01 17:08:26.087632 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:08:26.087585    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1b15a-predictor-84b9dd6865-v2nnf" event={"ID":"440dce6a-9f80-46d4-a092-9a34c40759cf","Type":"ContainerStarted","Data":"6ff19b0a4445987668088b4940fcb4a726966c272d6ee4e602eb3e522d7e3ca7"}
Jul 01 17:08:26.089403 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:08:26.089372    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1b15a-predictor-5c76b7cdd8-56vrx" event={"ID":"22a5235c-0a90-4a58-b636-3c53080e34c7","Type":"ContainerStarted","Data":"2c02e64049286e50cba58d10109dea330c2115f2885487a4c23d3ca0b34ca72a"}
Jul 01 17:08:29.105060 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:08:29.105028    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1b15a-predictor-84b9dd6865-v2nnf" event={"ID":"440dce6a-9f80-46d4-a092-9a34c40759cf","Type":"ContainerStarted","Data":"9c15eb0644f2a66532f921cce19ec539d0901bf3a4fdf09a93e10bda72e00c8a"}
Jul 01 17:08:29.105560 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:08:29.105161    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-1b15a-predictor-84b9dd6865-v2nnf"
Jul 01 17:08:29.106857 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:08:29.106832    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1b15a-predictor-5c76b7cdd8-56vrx" event={"ID":"22a5235c-0a90-4a58-b636-3c53080e34c7","Type":"ContainerStarted","Data":"5a84da4063aaa6ae7b46b35a4c0f1b642315400caec6c2bc79aa1224f9bc80b6"}
Jul 01 17:08:29.106997 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:08:29.106985    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-1b15a-predictor-5c76b7cdd8-56vrx"
Jul 01 17:08:29.107043 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:08:29.107007    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-1b15a-predictor-5c76b7cdd8-56vrx"
Jul 01 17:08:29.108469 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:08:29.108431    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1b15a-predictor-5c76b7cdd8-56vrx" podUID="22a5235c-0a90-4a58-b636-3c53080e34c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 01 17:08:29.132500 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:08:29.132432    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-1b15a-predictor-84b9dd6865-v2nnf" podStartSLOduration=1.6254455970000001 podStartE2EDuration="23.132412986s" totalImagesPullingTime="21.506967389s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:08:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:08:29.131627887 +0000 UTC m=+626.928766300" watchObservedRunningTime="2026-07-01 17:08:29.132412986 +0000 UTC m=+626.929551390"
Jul 01 17:08:29.150987 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:08:29.150938    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-1b15a-predictor-5c76b7cdd8-56vrx" podStartSLOduration=2.51796214 podStartE2EDuration="23.150922222s" totalImagesPullingTime="20.632960082s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:08:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:08:29.150461642 +0000 UTC m=+626.947600063" watchObservedRunningTime="2026-07-01 17:08:29.150922222 +0000 UTC m=+626.948060738"
Jul 01 17:08:30.111371 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:08:30.111282    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-1b15a-predictor-84b9dd6865-v2nnf"
Jul 01 17:08:30.111927 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:08:30.111886    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1b15a-predictor-5c76b7cdd8-56vrx" podUID="22a5235c-0a90-4a58-b636-3c53080e34c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 01 17:08:30.112686 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:08:30.112656    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1b15a-predictor-84b9dd6865-v2nnf" podUID="440dce6a-9f80-46d4-a092-9a34c40759cf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 01 17:08:31.115588 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:08:31.115544    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1b15a-predictor-84b9dd6865-v2nnf" podUID="440dce6a-9f80-46d4-a092-9a34c40759cf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 01 17:08:31.398381 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:08:31.398265    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:08:35.117517 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:08:35.117481    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-1b15a-predictor-5c76b7cdd8-56vrx"
Jul 01 17:08:35.118094 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:08:35.118063    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1b15a-predictor-5c76b7cdd8-56vrx" podUID="22a5235c-0a90-4a58-b636-3c53080e34c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 01 17:08:36.120556 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:08:36.120524    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-1b15a-predictor-84b9dd6865-v2nnf"
Jul 01 17:08:36.121042 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:08:36.121015    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1b15a-predictor-84b9dd6865-v2nnf" podUID="440dce6a-9f80-46d4-a092-9a34c40759cf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 01 17:08:42.960274 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:08:42.960232    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:08:43.031211 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:08:43.031172    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:08:45.118939 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:08:45.118892    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1b15a-predictor-5c76b7cdd8-56vrx" podUID="22a5235c-0a90-4a58-b636-3c53080e34c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 01 17:08:46.121462 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:08:46.121418    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1b15a-predictor-84b9dd6865-v2nnf" podUID="440dce6a-9f80-46d4-a092-9a34c40759cf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 01 17:08:55.118993 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:08:55.118951    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1b15a-predictor-5c76b7cdd8-56vrx" podUID="22a5235c-0a90-4a58-b636-3c53080e34c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 01 17:08:56.121216 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:08:56.121177    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1b15a-predictor-84b9dd6865-v2nnf" podUID="440dce6a-9f80-46d4-a092-9a34c40759cf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 01 17:09:05.118984 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:05.118942    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1b15a-predictor-5c76b7cdd8-56vrx" podUID="22a5235c-0a90-4a58-b636-3c53080e34c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 01 17:09:06.121323 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:06.121283    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1b15a-predictor-84b9dd6865-v2nnf" podUID="440dce6a-9f80-46d4-a092-9a34c40759cf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 01 17:09:15.118629 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:15.118597    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-1b15a-predictor-5c76b7cdd8-56vrx"
Jul 01 17:09:16.122576 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:16.122540    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-1b15a-predictor-84b9dd6865-v2nnf"
Jul 01 17:09:36.172963 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:36.172929    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-1b15a-5bf689b779-zll7z"]
Jul 01 17:09:36.176589 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:36.176571    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-1b15a-5bf689b779-zll7z"
Jul 01 17:09:36.179217 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:36.179199    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-1b15a-serving-cert\""
Jul 01 17:09:36.179328 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:36.179257    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-1b15a-kube-rbac-proxy-sar-config\""
Jul 01 17:09:36.185192 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:36.185168    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-1b15a-5bf689b779-zll7z"]
Jul 01 17:09:36.227048 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:36.227021    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:09:36.346956 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:36.346919    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d52c91d6-3f99-4d01-9226-ce261eb999b1-openshift-service-ca-bundle\") pod \"switch-graph-1b15a-5bf689b779-zll7z\" (UID: \"d52c91d6-3f99-4d01-9226-ce261eb999b1\") " pod="kserve-ci-e2e-test/switch-graph-1b15a-5bf689b779-zll7z"
Jul 01 17:09:36.347135 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:36.347025    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d52c91d6-3f99-4d01-9226-ce261eb999b1-proxy-tls\") pod \"switch-graph-1b15a-5bf689b779-zll7z\" (UID: \"d52c91d6-3f99-4d01-9226-ce261eb999b1\") " pod="kserve-ci-e2e-test/switch-graph-1b15a-5bf689b779-zll7z"
Jul 01 17:09:36.447897 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:36.447825    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d52c91d6-3f99-4d01-9226-ce261eb999b1-openshift-service-ca-bundle\") pod \"switch-graph-1b15a-5bf689b779-zll7z\" (UID: \"d52c91d6-3f99-4d01-9226-ce261eb999b1\") " pod="kserve-ci-e2e-test/switch-graph-1b15a-5bf689b779-zll7z"
Jul 01 17:09:36.448037 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:36.447896    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d52c91d6-3f99-4d01-9226-ce261eb999b1-proxy-tls\") pod \"switch-graph-1b15a-5bf689b779-zll7z\" (UID: \"d52c91d6-3f99-4d01-9226-ce261eb999b1\") " pod="kserve-ci-e2e-test/switch-graph-1b15a-5bf689b779-zll7z"
Jul 01 17:09:36.448037 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:09:36.447989    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-1b15a-serving-cert: secret "switch-graph-1b15a-serving-cert" not found
Jul 01 17:09:36.448147 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:09:36.448057    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d52c91d6-3f99-4d01-9226-ce261eb999b1-proxy-tls podName:d52c91d6-3f99-4d01-9226-ce261eb999b1 nodeName:}" failed. No retries permitted until 2026-07-01 17:09:36.94803736 +0000 UTC m=+694.745175757 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d52c91d6-3f99-4d01-9226-ce261eb999b1-proxy-tls") pod "switch-graph-1b15a-5bf689b779-zll7z" (UID: "d52c91d6-3f99-4d01-9226-ce261eb999b1") : secret "switch-graph-1b15a-serving-cert" not found
Jul 01 17:09:36.448494 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:36.448471    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d52c91d6-3f99-4d01-9226-ce261eb999b1-openshift-service-ca-bundle\") pod \"switch-graph-1b15a-5bf689b779-zll7z\" (UID: \"d52c91d6-3f99-4d01-9226-ce261eb999b1\") " pod="kserve-ci-e2e-test/switch-graph-1b15a-5bf689b779-zll7z"
Jul 01 17:09:36.559149 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:36.559109    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:09:36.951643 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:36.951593    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d52c91d6-3f99-4d01-9226-ce261eb999b1-proxy-tls\") pod \"switch-graph-1b15a-5bf689b779-zll7z\" (UID: \"d52c91d6-3f99-4d01-9226-ce261eb999b1\") " pod="kserve-ci-e2e-test/switch-graph-1b15a-5bf689b779-zll7z"
Jul 01 17:09:36.954179 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:36.954147    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d52c91d6-3f99-4d01-9226-ce261eb999b1-proxy-tls\") pod \"switch-graph-1b15a-5bf689b779-zll7z\" (UID: \"d52c91d6-3f99-4d01-9226-ce261eb999b1\") " pod="kserve-ci-e2e-test/switch-graph-1b15a-5bf689b779-zll7z"
Jul 01 17:09:37.087563 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:37.087528    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-1b15a-5bf689b779-zll7z"
Jul 01 17:09:37.217093 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:37.217065    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-1b15a-5bf689b779-zll7z"]
Jul 01 17:09:37.219999 ip-10-0-137-180 kubenswrapper[2563]: W0701 17:09:37.219964    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd52c91d6_3f99_4d01_9226_ce261eb999b1.slice/crio-0a3c18c00fb80e5c710440c906f5af61d918134a471c804bccb2c214bf54b758 WatchSource:0}: Error finding container 0a3c18c00fb80e5c710440c906f5af61d918134a471c804bccb2c214bf54b758: Status 404 returned error can't find the container with id 0a3c18c00fb80e5c710440c906f5af61d918134a471c804bccb2c214bf54b758
Jul 01 17:09:37.221910 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:37.221891    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 01 17:09:37.370099 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:37.370068    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-1b15a-5bf689b779-zll7z" event={"ID":"d52c91d6-3f99-4d01-9226-ce261eb999b1","Type":"ContainerStarted","Data":"0a3c18c00fb80e5c710440c906f5af61d918134a471c804bccb2c214bf54b758"}
Jul 01 17:09:40.384895 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:40.384858    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-1b15a-5bf689b779-zll7z" event={"ID":"d52c91d6-3f99-4d01-9226-ce261eb999b1","Type":"ContainerStarted","Data":"9b902510d03a129bb17edd823ca85c9a4baa216184441ddafc9172e4ac636187"}
Jul 01 17:09:40.385305 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:40.384979    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-1b15a-5bf689b779-zll7z"
Jul 01 17:09:40.402557 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:40.402511    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-1b15a-5bf689b779-zll7z" podStartSLOduration=1.784703427 podStartE2EDuration="4.40249876s" totalImagesPullingTime="2.617795333s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:09:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:09:40.401876993 +0000 UTC m=+698.199015393" watchObservedRunningTime="2026-07-01 17:09:40.40249876 +0000 UTC m=+698.199637139"
Jul 01 17:09:46.394913 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:46.394883    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-1b15a-5bf689b779-zll7z"
Jul 01 17:09:50.388171 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:50.388138    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-1b15a-5bf689b779-zll7z"]
Jul 01 17:09:50.388600 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:50.388377    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-1b15a-5bf689b779-zll7z" podUID="d52c91d6-3f99-4d01-9226-ce261eb999b1" containerName="switch-graph-1b15a" containerID="cri-o://9b902510d03a129bb17edd823ca85c9a4baa216184441ddafc9172e4ac636187" gracePeriod=30
Jul 01 17:09:50.413597 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:50.413558    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:09:50.548394 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:50.548338    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:09:50.588866 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:50.588832    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3c336-predictor-547bb6bc8f-96z6b"]
Jul 01 17:09:50.595542 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:50.595518    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3c336-predictor-547bb6bc8f-96z6b"
Jul 01 17:09:50.598267 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:50.598245    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-3c336-kube-rbac-proxy-sar-config\""
Jul 01 17:09:50.598445 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:50.598424    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-3c336-predictor-serving-cert\""
Jul 01 17:09:50.611334 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:50.611312    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3c336-predictor-547bb6bc8f-96z6b"]
Jul 01 17:09:50.629634 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:50.629600    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1b15a-predictor-84b9dd6865-v2nnf"]
Jul 01 17:09:50.629847 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:50.629824    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-1b15a-predictor-84b9dd6865-v2nnf" podUID="440dce6a-9f80-46d4-a092-9a34c40759cf" containerName="kserve-container" containerID="cri-o://6ff19b0a4445987668088b4940fcb4a726966c272d6ee4e602eb3e522d7e3ca7" gracePeriod=30
Jul 01 17:09:50.629908 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:50.629892    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-1b15a-predictor-84b9dd6865-v2nnf" podUID="440dce6a-9f80-46d4-a092-9a34c40759cf" containerName="kube-rbac-proxy" containerID="cri-o://9c15eb0644f2a66532f921cce19ec539d0901bf3a4fdf09a93e10bda72e00c8a" gracePeriod=30
Jul 01 17:09:50.651054 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:50.651025    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:09:50.664050 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:50.664020    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc760277-8744-4acb-89dc-4cbc32f83742-proxy-tls\") pod \"success-200-isvc-3c336-predictor-547bb6bc8f-96z6b\" (UID: \"dc760277-8744-4acb-89dc-4cbc32f83742\") " pod="kserve-ci-e2e-test/success-200-isvc-3c336-predictor-547bb6bc8f-96z6b"
Jul 01 17:09:50.664410 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:50.664376    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc760277-8744-4acb-89dc-4cbc32f83742-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-3c336-predictor-547bb6bc8f-96z6b\" (UID: \"dc760277-8744-4acb-89dc-4cbc32f83742\") " pod="kserve-ci-e2e-test/success-200-isvc-3c336-predictor-547bb6bc8f-96z6b"
Jul 01 17:09:50.664508 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:50.664443    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx7nz\" (UniqueName: \"kubernetes.io/projected/dc760277-8744-4acb-89dc-4cbc32f83742-kube-api-access-sx7nz\") pod \"success-200-isvc-3c336-predictor-547bb6bc8f-96z6b\" (UID: \"dc760277-8744-4acb-89dc-4cbc32f83742\") " pod="kserve-ci-e2e-test/success-200-isvc-3c336-predictor-547bb6bc8f-96z6b"
Jul 01 17:09:50.677095 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:50.677064    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3c336-predictor-54665d447b-s2vc2"]
Jul 01 17:09:50.685477 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:50.685262    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3c336-predictor-54665d447b-s2vc2"
Jul 01 17:09:50.688375 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:50.688291    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-3c336-kube-rbac-proxy-sar-config\""
Jul 01 17:09:50.688375 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:50.688308    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-3c336-predictor-serving-cert\""
Jul 01 17:09:50.699301 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:50.699277    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3c336-predictor-54665d447b-s2vc2"]
Jul 01 17:09:50.745845 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:50.745810    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:09:50.746110 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:50.746091    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1b15a-predictor-5c76b7cdd8-56vrx"]
Jul 01 17:09:50.746385 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:50.746327    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-1b15a-predictor-5c76b7cdd8-56vrx" podUID="22a5235c-0a90-4a58-b636-3c53080e34c7" containerName="kserve-container" containerID="cri-o://2c02e64049286e50cba58d10109dea330c2115f2885487a4c23d3ca0b34ca72a" gracePeriod=30
Jul 01 17:09:50.746521 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:50.746416    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-1b15a-predictor-5c76b7cdd8-56vrx" podUID="22a5235c-0a90-4a58-b636-3c53080e34c7" containerName="kube-rbac-proxy" containerID="cri-o://5a84da4063aaa6ae7b46b35a4c0f1b642315400caec6c2bc79aa1224f9bc80b6" gracePeriod=30
Jul 01 17:09:50.765372 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:50.765331    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx7nz\" (UniqueName: \"kubernetes.io/projected/dc760277-8744-4acb-89dc-4cbc32f83742-kube-api-access-sx7nz\") pod \"success-200-isvc-3c336-predictor-547bb6bc8f-96z6b\" (UID: \"dc760277-8744-4acb-89dc-4cbc32f83742\") " pod="kserve-ci-e2e-test/success-200-isvc-3c336-predictor-547bb6bc8f-96z6b"
Jul 01 17:09:50.765469 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:50.765421    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0ce82ad2-7236-475b-b3ff-eccfa49eccfa-proxy-tls\") pod \"error-404-isvc-3c336-predictor-54665d447b-s2vc2\" (UID: \"0ce82ad2-7236-475b-b3ff-eccfa49eccfa\") " pod="kserve-ci-e2e-test/error-404-isvc-3c336-predictor-54665d447b-s2vc2"
Jul 01 17:09:50.765534 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:50.765472    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6xtq\" (UniqueName: \"kubernetes.io/projected/0ce82ad2-7236-475b-b3ff-eccfa49eccfa-kube-api-access-m6xtq\") pod \"error-404-isvc-3c336-predictor-54665d447b-s2vc2\" (UID: \"0ce82ad2-7236-475b-b3ff-eccfa49eccfa\") " pod="kserve-ci-e2e-test/error-404-isvc-3c336-predictor-54665d447b-s2vc2"
Jul 01 17:09:50.765617 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:50.765589    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0ce82ad2-7236-475b-b3ff-eccfa49eccfa-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-3c336-predictor-54665d447b-s2vc2\" (UID: \"0ce82ad2-7236-475b-b3ff-eccfa49eccfa\") " pod="kserve-ci-e2e-test/error-404-isvc-3c336-predictor-54665d447b-s2vc2"
Jul 01 17:09:50.765666 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:50.765623    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc760277-8744-4acb-89dc-4cbc32f83742-proxy-tls\") pod \"success-200-isvc-3c336-predictor-547bb6bc8f-96z6b\" (UID: \"dc760277-8744-4acb-89dc-4cbc32f83742\") " pod="kserve-ci-e2e-test/success-200-isvc-3c336-predictor-547bb6bc8f-96z6b"
Jul 01 17:09:50.765702 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:50.765681    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc760277-8744-4acb-89dc-4cbc32f83742-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-3c336-predictor-547bb6bc8f-96z6b\" (UID: \"dc760277-8744-4acb-89dc-4cbc32f83742\") " pod="kserve-ci-e2e-test/success-200-isvc-3c336-predictor-547bb6bc8f-96z6b"
Jul 01 17:09:50.765838 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:09:50.765802    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-3c336-predictor-serving-cert: secret "success-200-isvc-3c336-predictor-serving-cert" not found
Jul 01 17:09:50.765932 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:09:50.765874    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc760277-8744-4acb-89dc-4cbc32f83742-proxy-tls podName:dc760277-8744-4acb-89dc-4cbc32f83742 nodeName:}" failed. No retries permitted until 2026-07-01 17:09:51.265853074 +0000 UTC m=+709.062991467 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/dc760277-8744-4acb-89dc-4cbc32f83742-proxy-tls") pod "success-200-isvc-3c336-predictor-547bb6bc8f-96z6b" (UID: "dc760277-8744-4acb-89dc-4cbc32f83742") : secret "success-200-isvc-3c336-predictor-serving-cert" not found
Jul 01 17:09:50.766433 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:50.766389    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc760277-8744-4acb-89dc-4cbc32f83742-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-3c336-predictor-547bb6bc8f-96z6b\" (UID: \"dc760277-8744-4acb-89dc-4cbc32f83742\") " pod="kserve-ci-e2e-test/success-200-isvc-3c336-predictor-547bb6bc8f-96z6b"
Jul 01 17:09:50.774935 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:50.774914    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx7nz\" (UniqueName: \"kubernetes.io/projected/dc760277-8744-4acb-89dc-4cbc32f83742-kube-api-access-sx7nz\") pod \"success-200-isvc-3c336-predictor-547bb6bc8f-96z6b\" (UID: \"dc760277-8744-4acb-89dc-4cbc32f83742\") " pod="kserve-ci-e2e-test/success-200-isvc-3c336-predictor-547bb6bc8f-96z6b"
Jul 01 17:09:50.846642 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:50.846609    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:09:50.866794 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:50.866768    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0ce82ad2-7236-475b-b3ff-eccfa49eccfa-proxy-tls\") pod \"error-404-isvc-3c336-predictor-54665d447b-s2vc2\" (UID: \"0ce82ad2-7236-475b-b3ff-eccfa49eccfa\") " pod="kserve-ci-e2e-test/error-404-isvc-3c336-predictor-54665d447b-s2vc2"
Jul 01 17:09:50.866895 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:50.866827    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6xtq\" (UniqueName: \"kubernetes.io/projected/0ce82ad2-7236-475b-b3ff-eccfa49eccfa-kube-api-access-m6xtq\") pod \"error-404-isvc-3c336-predictor-54665d447b-s2vc2\" (UID: \"0ce82ad2-7236-475b-b3ff-eccfa49eccfa\") " pod="kserve-ci-e2e-test/error-404-isvc-3c336-predictor-54665d447b-s2vc2"
Jul 01 17:09:50.866895 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:50.866858    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0ce82ad2-7236-475b-b3ff-eccfa49eccfa-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-3c336-predictor-54665d447b-s2vc2\" (UID: \"0ce82ad2-7236-475b-b3ff-eccfa49eccfa\") " pod="kserve-ci-e2e-test/error-404-isvc-3c336-predictor-54665d447b-s2vc2"
Jul 01 17:09:50.867589 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:50.867568    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0ce82ad2-7236-475b-b3ff-eccfa49eccfa-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-3c336-predictor-54665d447b-s2vc2\" (UID: \"0ce82ad2-7236-475b-b3ff-eccfa49eccfa\") " pod="kserve-ci-e2e-test/error-404-isvc-3c336-predictor-54665d447b-s2vc2"
Jul 01 17:09:50.869237 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:50.869218    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0ce82ad2-7236-475b-b3ff-eccfa49eccfa-proxy-tls\") pod \"error-404-isvc-3c336-predictor-54665d447b-s2vc2\" (UID: \"0ce82ad2-7236-475b-b3ff-eccfa49eccfa\") " pod="kserve-ci-e2e-test/error-404-isvc-3c336-predictor-54665d447b-s2vc2"
Jul 01 17:09:50.876220 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:50.876195    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6xtq\" (UniqueName: \"kubernetes.io/projected/0ce82ad2-7236-475b-b3ff-eccfa49eccfa-kube-api-access-m6xtq\") pod \"error-404-isvc-3c336-predictor-54665d447b-s2vc2\" (UID: \"0ce82ad2-7236-475b-b3ff-eccfa49eccfa\") " pod="kserve-ci-e2e-test/error-404-isvc-3c336-predictor-54665d447b-s2vc2"
Jul 01 17:09:50.945528 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:50.945448    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:09:50.999717 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:50.999677    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3c336-predictor-54665d447b-s2vc2"
Jul 01 17:09:51.044988 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:51.044955    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:09:51.115991 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:51.115945    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1b15a-predictor-84b9dd6865-v2nnf" podUID="440dce6a-9f80-46d4-a092-9a34c40759cf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.23:8643/healthz\": dial tcp 10.132.0.23:8643: connect: connection refused"
Jul 01 17:09:51.129791 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:51.129635    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3c336-predictor-54665d447b-s2vc2"]
Jul 01 17:09:51.132440 ip-10-0-137-180 kubenswrapper[2563]: W0701 17:09:51.132417    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ce82ad2_7236_475b_b3ff_eccfa49eccfa.slice/crio-edc723769dc8f39359aecc8c3f2ec68e2820f24daee3a2bcab24bd2fda8e984a WatchSource:0}: Error finding container edc723769dc8f39359aecc8c3f2ec68e2820f24daee3a2bcab24bd2fda8e984a: Status 404 returned error can't find the container with id edc723769dc8f39359aecc8c3f2ec68e2820f24daee3a2bcab24bd2fda8e984a
Jul 01 17:09:51.269781 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:51.269753    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc760277-8744-4acb-89dc-4cbc32f83742-proxy-tls\") pod \"success-200-isvc-3c336-predictor-547bb6bc8f-96z6b\" (UID: \"dc760277-8744-4acb-89dc-4cbc32f83742\") " pod="kserve-ci-e2e-test/success-200-isvc-3c336-predictor-547bb6bc8f-96z6b"
Jul 01 17:09:51.272217 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:51.272195    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc760277-8744-4acb-89dc-4cbc32f83742-proxy-tls\") pod \"success-200-isvc-3c336-predictor-547bb6bc8f-96z6b\" (UID: \"dc760277-8744-4acb-89dc-4cbc32f83742\") " pod="kserve-ci-e2e-test/success-200-isvc-3c336-predictor-547bb6bc8f-96z6b"
Jul 01 17:09:51.393869 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:51.393826    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-1b15a-5bf689b779-zll7z" podUID="d52c91d6-3f99-4d01-9226-ce261eb999b1" containerName="switch-graph-1b15a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:09:51.427277 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:51.427235    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3c336-predictor-54665d447b-s2vc2" event={"ID":"0ce82ad2-7236-475b-b3ff-eccfa49eccfa","Type":"ContainerStarted","Data":"0b7ad5277b8472209ebfc568435438f7aa1fb5739fc8d1f25a128c2a020f59b1"}
Jul 01 17:09:51.427486 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:51.427297    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3c336-predictor-54665d447b-s2vc2" event={"ID":"0ce82ad2-7236-475b-b3ff-eccfa49eccfa","Type":"ContainerStarted","Data":"624b66f1b669b1173df86310452a5c315ded3f96a559c446c912fd16e2a1723b"}
Jul 01 17:09:51.427486 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:51.427452    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-3c336-predictor-54665d447b-s2vc2"
Jul 01 17:09:51.427600 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:51.427491    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3c336-predictor-54665d447b-s2vc2" event={"ID":"0ce82ad2-7236-475b-b3ff-eccfa49eccfa","Type":"ContainerStarted","Data":"edc723769dc8f39359aecc8c3f2ec68e2820f24daee3a2bcab24bd2fda8e984a"}
Jul 01 17:09:51.427600 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:51.427520    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-3c336-predictor-54665d447b-s2vc2"
Jul 01 17:09:51.429107 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:51.429082    2563 generic.go:356] "Generic (PLEG): container finished" podID="440dce6a-9f80-46d4-a092-9a34c40759cf" containerID="9c15eb0644f2a66532f921cce19ec539d0901bf3a4fdf09a93e10bda72e00c8a" exitCode=2
Jul 01 17:09:51.429107 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:51.429093    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3c336-predictor-54665d447b-s2vc2" podUID="0ce82ad2-7236-475b-b3ff-eccfa49eccfa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 01 17:09:51.429293 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:51.429153    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1b15a-predictor-84b9dd6865-v2nnf" event={"ID":"440dce6a-9f80-46d4-a092-9a34c40759cf","Type":"ContainerDied","Data":"9c15eb0644f2a66532f921cce19ec539d0901bf3a4fdf09a93e10bda72e00c8a"}
Jul 01 17:09:51.430876 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:51.430848    2563 generic.go:356] "Generic (PLEG): container finished" podID="22a5235c-0a90-4a58-b636-3c53080e34c7" containerID="5a84da4063aaa6ae7b46b35a4c0f1b642315400caec6c2bc79aa1224f9bc80b6" exitCode=2
Jul 01 17:09:51.430979 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:51.430878    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1b15a-predictor-5c76b7cdd8-56vrx" event={"ID":"22a5235c-0a90-4a58-b636-3c53080e34c7","Type":"ContainerDied","Data":"5a84da4063aaa6ae7b46b35a4c0f1b642315400caec6c2bc79aa1224f9bc80b6"}
Jul 01 17:09:51.445845 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:51.445796    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-3c336-predictor-54665d447b-s2vc2" podStartSLOduration=1.445782497 podStartE2EDuration="1.445782497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:09:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:09:51.444515147 +0000 UTC m=+709.241653565" watchObservedRunningTime="2026-07-01 17:09:51.445782497 +0000 UTC m=+709.242920898"
Jul 01 17:09:51.508723 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:51.508685    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3c336-predictor-547bb6bc8f-96z6b"
Jul 01 17:09:51.648364 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:51.648280    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3c336-predictor-547bb6bc8f-96z6b"]
Jul 01 17:09:51.651718 ip-10-0-137-180 kubenswrapper[2563]: W0701 17:09:51.651691    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc760277_8744_4acb_89dc_4cbc32f83742.slice/crio-a9341343c538a35b8a36009181027ece843b15801c84737b604b28cd9ac725e1 WatchSource:0}: Error finding container a9341343c538a35b8a36009181027ece843b15801c84737b604b28cd9ac725e1: Status 404 returned error can't find the container with id a9341343c538a35b8a36009181027ece843b15801c84737b604b28cd9ac725e1
Jul 01 17:09:52.436512 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:52.436481    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3c336-predictor-547bb6bc8f-96z6b" event={"ID":"dc760277-8744-4acb-89dc-4cbc32f83742","Type":"ContainerStarted","Data":"c0d752993c86d6c7a8d349f2444a6d7030def616cbc34f7cb85a847cd7ada222"}
Jul 01 17:09:52.436927 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:52.436536    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3c336-predictor-547bb6bc8f-96z6b" event={"ID":"dc760277-8744-4acb-89dc-4cbc32f83742","Type":"ContainerStarted","Data":"9f335543f9b6ea288b41e81269025946946c16bb316bad568b859148f62e1686"}
Jul 01 17:09:52.436927 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:52.436561    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3c336-predictor-547bb6bc8f-96z6b" event={"ID":"dc760277-8744-4acb-89dc-4cbc32f83742","Type":"ContainerStarted","Data":"a9341343c538a35b8a36009181027ece843b15801c84737b604b28cd9ac725e1"}
Jul 01 17:09:52.436927 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:52.436700    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-3c336-predictor-547bb6bc8f-96z6b"
Jul 01 17:09:52.436927 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:52.436746    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-3c336-predictor-547bb6bc8f-96z6b"
Jul 01 17:09:52.436927 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:52.436903    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3c336-predictor-54665d447b-s2vc2" podUID="0ce82ad2-7236-475b-b3ff-eccfa49eccfa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 01 17:09:52.437866 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:52.437847    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3c336-predictor-547bb6bc8f-96z6b" podUID="dc760277-8744-4acb-89dc-4cbc32f83742" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 01 17:09:52.454113 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:52.454059    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-3c336-predictor-547bb6bc8f-96z6b" podStartSLOduration=2.454045572 podStartE2EDuration="2.454045572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:09:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:09:52.453545459 +0000 UTC m=+710.250683873" watchObservedRunningTime="2026-07-01 17:09:52.454045572 +0000 UTC m=+710.251183972"
Jul 01 17:09:53.440758 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:53.440709    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3c336-predictor-547bb6bc8f-96z6b" podUID="dc760277-8744-4acb-89dc-4cbc32f83742" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 01 17:09:53.982047 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:53.982023    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-1b15a-predictor-84b9dd6865-v2nnf"
Jul 01 17:09:54.092789 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:54.092695    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/440dce6a-9f80-46d4-a092-9a34c40759cf-kube-rbac-proxy-sar-config\") pod \"440dce6a-9f80-46d4-a092-9a34c40759cf\" (UID: \"440dce6a-9f80-46d4-a092-9a34c40759cf\") "
Jul 01 17:09:54.092940 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:54.092842    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-49h79\" (UniqueName: \"kubernetes.io/projected/440dce6a-9f80-46d4-a092-9a34c40759cf-kube-api-access-49h79\") pod \"440dce6a-9f80-46d4-a092-9a34c40759cf\" (UID: \"440dce6a-9f80-46d4-a092-9a34c40759cf\") "
Jul 01 17:09:54.092940 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:54.092873    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/440dce6a-9f80-46d4-a092-9a34c40759cf-proxy-tls\") pod \"440dce6a-9f80-46d4-a092-9a34c40759cf\" (UID: \"440dce6a-9f80-46d4-a092-9a34c40759cf\") "
Jul 01 17:09:54.093118 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:54.093088    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/440dce6a-9f80-46d4-a092-9a34c40759cf-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "440dce6a-9f80-46d4-a092-9a34c40759cf" (UID: "440dce6a-9f80-46d4-a092-9a34c40759cf"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:09:54.095179 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:54.095148    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/440dce6a-9f80-46d4-a092-9a34c40759cf-kube-api-access-49h79" (OuterVolumeSpecName: "kube-api-access-49h79") pod "440dce6a-9f80-46d4-a092-9a34c40759cf" (UID: "440dce6a-9f80-46d4-a092-9a34c40759cf"). InnerVolumeSpecName "kube-api-access-49h79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 17:09:54.095179 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:54.095168    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/440dce6a-9f80-46d4-a092-9a34c40759cf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "440dce6a-9f80-46d4-a092-9a34c40759cf" (UID: "440dce6a-9f80-46d4-a092-9a34c40759cf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 17:09:54.179516 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:54.179492    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1b15a-predictor-5c76b7cdd8-56vrx"
Jul 01 17:09:54.194120 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:54.194096    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49h79\" (UniqueName: \"kubernetes.io/projected/440dce6a-9f80-46d4-a092-9a34c40759cf-kube-api-access-49h79\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:09:54.194120 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:54.194116    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/440dce6a-9f80-46d4-a092-9a34c40759cf-proxy-tls\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:09:54.194251 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:54.194126    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/440dce6a-9f80-46d4-a092-9a34c40759cf-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:09:54.294970 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:54.294939    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22a5235c-0a90-4a58-b636-3c53080e34c7-kube-rbac-proxy-sar-config\") pod \"22a5235c-0a90-4a58-b636-3c53080e34c7\" (UID: \"22a5235c-0a90-4a58-b636-3c53080e34c7\") "
Jul 01 17:09:54.295106 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:54.295004    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22a5235c-0a90-4a58-b636-3c53080e34c7-proxy-tls\") pod \"22a5235c-0a90-4a58-b636-3c53080e34c7\" (UID: \"22a5235c-0a90-4a58-b636-3c53080e34c7\") "
Jul 01 17:09:54.295106 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:54.295039    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mmctr\" (UniqueName: \"kubernetes.io/projected/22a5235c-0a90-4a58-b636-3c53080e34c7-kube-api-access-mmctr\") pod \"22a5235c-0a90-4a58-b636-3c53080e34c7\" (UID: \"22a5235c-0a90-4a58-b636-3c53080e34c7\") "
Jul 01 17:09:54.295364 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:54.295327    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22a5235c-0a90-4a58-b636-3c53080e34c7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "22a5235c-0a90-4a58-b636-3c53080e34c7" (UID: "22a5235c-0a90-4a58-b636-3c53080e34c7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:09:54.297139 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:54.297116    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22a5235c-0a90-4a58-b636-3c53080e34c7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "22a5235c-0a90-4a58-b636-3c53080e34c7" (UID: "22a5235c-0a90-4a58-b636-3c53080e34c7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 17:09:54.297226 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:54.297138    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22a5235c-0a90-4a58-b636-3c53080e34c7-kube-api-access-mmctr" (OuterVolumeSpecName: "kube-api-access-mmctr") pod "22a5235c-0a90-4a58-b636-3c53080e34c7" (UID: "22a5235c-0a90-4a58-b636-3c53080e34c7"). InnerVolumeSpecName "kube-api-access-mmctr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 17:09:54.396303 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:54.396222    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22a5235c-0a90-4a58-b636-3c53080e34c7-proxy-tls\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:09:54.396303 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:54.396250    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmctr\" (UniqueName: \"kubernetes.io/projected/22a5235c-0a90-4a58-b636-3c53080e34c7-kube-api-access-mmctr\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:09:54.396303 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:54.396261    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22a5235c-0a90-4a58-b636-3c53080e34c7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:09:54.446108 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:54.446073    2563 generic.go:356] "Generic (PLEG): container finished" podID="22a5235c-0a90-4a58-b636-3c53080e34c7" containerID="2c02e64049286e50cba58d10109dea330c2115f2885487a4c23d3ca0b34ca72a" exitCode=0
Jul 01 17:09:54.446514 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:54.446118    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1b15a-predictor-5c76b7cdd8-56vrx"
Jul 01 17:09:54.446514 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:54.446122    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1b15a-predictor-5c76b7cdd8-56vrx" event={"ID":"22a5235c-0a90-4a58-b636-3c53080e34c7","Type":"ContainerDied","Data":"2c02e64049286e50cba58d10109dea330c2115f2885487a4c23d3ca0b34ca72a"}
Jul 01 17:09:54.446514 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:54.446213    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1b15a-predictor-5c76b7cdd8-56vrx" event={"ID":"22a5235c-0a90-4a58-b636-3c53080e34c7","Type":"ContainerDied","Data":"90394b9b49ee3904fda592277edeee7005ffaa7d35b158ad33939b662843b6e3"}
Jul 01 17:09:54.446514 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:54.446243    2563 scope.go:117] "RemoveContainer" containerID="5a84da4063aaa6ae7b46b35a4c0f1b642315400caec6c2bc79aa1224f9bc80b6"
Jul 01 17:09:54.448046 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:54.448021    2563 generic.go:356] "Generic (PLEG): container finished" podID="440dce6a-9f80-46d4-a092-9a34c40759cf" containerID="6ff19b0a4445987668088b4940fcb4a726966c272d6ee4e602eb3e522d7e3ca7" exitCode=0
Jul 01 17:09:54.448235 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:54.448061    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1b15a-predictor-84b9dd6865-v2nnf" event={"ID":"440dce6a-9f80-46d4-a092-9a34c40759cf","Type":"ContainerDied","Data":"6ff19b0a4445987668088b4940fcb4a726966c272d6ee4e602eb3e522d7e3ca7"}
Jul 01 17:09:54.448235 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:54.448070    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-1b15a-predictor-84b9dd6865-v2nnf"
Jul 01 17:09:54.448235 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:54.448109    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1b15a-predictor-84b9dd6865-v2nnf" event={"ID":"440dce6a-9f80-46d4-a092-9a34c40759cf","Type":"ContainerDied","Data":"7119fea76ce34a57fbf6cb53ea097c66e20cc1193cccf4cca6c44732820dac7e"}
Jul 01 17:09:54.454978 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:54.454958    2563 scope.go:117] "RemoveContainer" containerID="2c02e64049286e50cba58d10109dea330c2115f2885487a4c23d3ca0b34ca72a"
Jul 01 17:09:54.462465 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:54.462450    2563 scope.go:117] "RemoveContainer" containerID="5a84da4063aaa6ae7b46b35a4c0f1b642315400caec6c2bc79aa1224f9bc80b6"
Jul 01 17:09:54.462688 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:09:54.462672    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a84da4063aaa6ae7b46b35a4c0f1b642315400caec6c2bc79aa1224f9bc80b6\": container with ID starting with 5a84da4063aaa6ae7b46b35a4c0f1b642315400caec6c2bc79aa1224f9bc80b6 not found: ID does not exist" containerID="5a84da4063aaa6ae7b46b35a4c0f1b642315400caec6c2bc79aa1224f9bc80b6"
Jul 01 17:09:54.462735 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:54.462695    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a84da4063aaa6ae7b46b35a4c0f1b642315400caec6c2bc79aa1224f9bc80b6"} err="failed to get container status \"5a84da4063aaa6ae7b46b35a4c0f1b642315400caec6c2bc79aa1224f9bc80b6\": rpc error: code = NotFound desc = could not find container \"5a84da4063aaa6ae7b46b35a4c0f1b642315400caec6c2bc79aa1224f9bc80b6\": container with ID starting with 5a84da4063aaa6ae7b46b35a4c0f1b642315400caec6c2bc79aa1224f9bc80b6 not found: ID does not exist"
Jul 01 17:09:54.462735 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:54.462710    2563 scope.go:117] "RemoveContainer" containerID="2c02e64049286e50cba58d10109dea330c2115f2885487a4c23d3ca0b34ca72a"
Jul 01 17:09:54.462942 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:09:54.462923    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c02e64049286e50cba58d10109dea330c2115f2885487a4c23d3ca0b34ca72a\": container with ID starting with 2c02e64049286e50cba58d10109dea330c2115f2885487a4c23d3ca0b34ca72a not found: ID does not exist" containerID="2c02e64049286e50cba58d10109dea330c2115f2885487a4c23d3ca0b34ca72a"
Jul 01 17:09:54.462999 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:54.462951    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c02e64049286e50cba58d10109dea330c2115f2885487a4c23d3ca0b34ca72a"} err="failed to get container status \"2c02e64049286e50cba58d10109dea330c2115f2885487a4c23d3ca0b34ca72a\": rpc error: code = NotFound desc = could not find container \"2c02e64049286e50cba58d10109dea330c2115f2885487a4c23d3ca0b34ca72a\": container with ID starting with 2c02e64049286e50cba58d10109dea330c2115f2885487a4c23d3ca0b34ca72a not found: ID does not exist"
Jul 01 17:09:54.462999 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:54.462974    2563 scope.go:117] "RemoveContainer" containerID="9c15eb0644f2a66532f921cce19ec539d0901bf3a4fdf09a93e10bda72e00c8a"
Jul 01 17:09:54.469064 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:54.469043    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1b15a-predictor-5c76b7cdd8-56vrx"]
Jul 01 17:09:54.471316 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:54.471297    2563 scope.go:117] "RemoveContainer" containerID="6ff19b0a4445987668088b4940fcb4a726966c272d6ee4e602eb3e522d7e3ca7"
Jul 01 17:09:54.472862 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:54.472841    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1b15a-predictor-5c76b7cdd8-56vrx"]
Jul 01 17:09:54.478511 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:54.478495    2563 scope.go:117] "RemoveContainer" containerID="9c15eb0644f2a66532f921cce19ec539d0901bf3a4fdf09a93e10bda72e00c8a"
Jul 01 17:09:54.478797 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:09:54.478780    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c15eb0644f2a66532f921cce19ec539d0901bf3a4fdf09a93e10bda72e00c8a\": container with ID starting with 9c15eb0644f2a66532f921cce19ec539d0901bf3a4fdf09a93e10bda72e00c8a not found: ID does not exist" containerID="9c15eb0644f2a66532f921cce19ec539d0901bf3a4fdf09a93e10bda72e00c8a"
Jul 01 17:09:54.478855 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:54.478800    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c15eb0644f2a66532f921cce19ec539d0901bf3a4fdf09a93e10bda72e00c8a"} err="failed to get container status \"9c15eb0644f2a66532f921cce19ec539d0901bf3a4fdf09a93e10bda72e00c8a\": rpc error: code = NotFound desc = could not find container \"9c15eb0644f2a66532f921cce19ec539d0901bf3a4fdf09a93e10bda72e00c8a\": container with ID starting with 9c15eb0644f2a66532f921cce19ec539d0901bf3a4fdf09a93e10bda72e00c8a not found: ID does not exist"
Jul 01 17:09:54.478855 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:54.478817    2563 scope.go:117] "RemoveContainer" containerID="6ff19b0a4445987668088b4940fcb4a726966c272d6ee4e602eb3e522d7e3ca7"
Jul 01 17:09:54.479013 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:09:54.479000    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ff19b0a4445987668088b4940fcb4a726966c272d6ee4e602eb3e522d7e3ca7\": container with ID starting with 6ff19b0a4445987668088b4940fcb4a726966c272d6ee4e602eb3e522d7e3ca7 not found: ID does not exist" containerID="6ff19b0a4445987668088b4940fcb4a726966c272d6ee4e602eb3e522d7e3ca7"
Jul 01 17:09:54.479057 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:54.479015    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ff19b0a4445987668088b4940fcb4a726966c272d6ee4e602eb3e522d7e3ca7"} err="failed to get container status \"6ff19b0a4445987668088b4940fcb4a726966c272d6ee4e602eb3e522d7e3ca7\": rpc error: code = NotFound desc = could not find container \"6ff19b0a4445987668088b4940fcb4a726966c272d6ee4e602eb3e522d7e3ca7\": container with ID starting with 6ff19b0a4445987668088b4940fcb4a726966c272d6ee4e602eb3e522d7e3ca7 not found: ID does not exist"
Jul 01 17:09:54.484325 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:54.484306    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1b15a-predictor-84b9dd6865-v2nnf"]
Jul 01 17:09:54.489802 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:54.489778    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1b15a-predictor-84b9dd6865-v2nnf"]
Jul 01 17:09:54.697004 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:54.696920    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22a5235c-0a90-4a58-b636-3c53080e34c7" path="/var/lib/kubelet/pods/22a5235c-0a90-4a58-b636-3c53080e34c7/volumes"
Jul 01 17:09:54.697370 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:54.697336    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="440dce6a-9f80-46d4-a092-9a34c40759cf" path="/var/lib/kubelet/pods/440dce6a-9f80-46d4-a092-9a34c40759cf/volumes"
Jul 01 17:09:56.393214 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:56.393175    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-1b15a-5bf689b779-zll7z" podUID="d52c91d6-3f99-4d01-9226-ce261eb999b1" containerName="switch-graph-1b15a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:09:57.441132 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:57.441105    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-3c336-predictor-54665d447b-s2vc2"
Jul 01 17:09:57.441602 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:57.441577    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3c336-predictor-54665d447b-s2vc2" podUID="0ce82ad2-7236-475b-b3ff-eccfa49eccfa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 01 17:09:58.446578 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:58.446547    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-3c336-predictor-547bb6bc8f-96z6b"
Jul 01 17:09:58.447441 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:09:58.447413    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3c336-predictor-547bb6bc8f-96z6b" podUID="dc760277-8744-4acb-89dc-4cbc32f83742" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 01 17:10:01.393661 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:01.393623    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-1b15a-5bf689b779-zll7z" podUID="d52c91d6-3f99-4d01-9226-ce261eb999b1" containerName="switch-graph-1b15a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:10:01.394038 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:01.393750    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-1b15a-5bf689b779-zll7z"
Jul 01 17:10:06.161251 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:06.161205    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:10:06.393191 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:06.393156    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-1b15a-5bf689b779-zll7z" podUID="d52c91d6-3f99-4d01-9226-ce261eb999b1" containerName="switch-graph-1b15a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:10:07.441838 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:07.441800    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3c336-predictor-54665d447b-s2vc2" podUID="0ce82ad2-7236-475b-b3ff-eccfa49eccfa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 01 17:10:07.574988 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:07.574950    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:10:08.448173 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:08.448131    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3c336-predictor-547bb6bc8f-96z6b" podUID="dc760277-8744-4acb-89dc-4cbc32f83742" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 01 17:10:11.393401 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:11.393330    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-1b15a-5bf689b779-zll7z" podUID="d52c91d6-3f99-4d01-9226-ce261eb999b1" containerName="switch-graph-1b15a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:10:16.299066 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:16.299031    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:10:16.392850 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:16.392808    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-1b15a-5bf689b779-zll7z" podUID="d52c91d6-3f99-4d01-9226-ce261eb999b1" containerName="switch-graph-1b15a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:10:16.467203 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:16.467156    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:10:16.538433 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:16.538399    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ce016-predictor-6ffdb68b79-hq49w"]
Jul 01 17:10:16.538848 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:16.538822    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22a5235c-0a90-4a58-b636-3c53080e34c7" containerName="kserve-container"
Jul 01 17:10:16.538848 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:16.538848    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="22a5235c-0a90-4a58-b636-3c53080e34c7" containerName="kserve-container"
Jul 01 17:10:16.539033 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:16.538871    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22a5235c-0a90-4a58-b636-3c53080e34c7" containerName="kube-rbac-proxy"
Jul 01 17:10:16.539033 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:16.538880    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="22a5235c-0a90-4a58-b636-3c53080e34c7" containerName="kube-rbac-proxy"
Jul 01 17:10:16.539033 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:16.538892    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="440dce6a-9f80-46d4-a092-9a34c40759cf" containerName="kserve-container"
Jul 01 17:10:16.539033 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:16.538900    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="440dce6a-9f80-46d4-a092-9a34c40759cf" containerName="kserve-container"
Jul 01 17:10:16.539033 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:16.538912    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="440dce6a-9f80-46d4-a092-9a34c40759cf" containerName="kube-rbac-proxy"
Jul 01 17:10:16.539033 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:16.538921    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="440dce6a-9f80-46d4-a092-9a34c40759cf" containerName="kube-rbac-proxy"
Jul 01 17:10:16.539033 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:16.539009    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="22a5235c-0a90-4a58-b636-3c53080e34c7" containerName="kube-rbac-proxy"
Jul 01 17:10:16.539033 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:16.539022    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="440dce6a-9f80-46d4-a092-9a34c40759cf" containerName="kube-rbac-proxy"
Jul 01 17:10:16.539033 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:16.539032    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="440dce6a-9f80-46d4-a092-9a34c40759cf" containerName="kserve-container"
Jul 01 17:10:16.539477 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:16.539042    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="22a5235c-0a90-4a58-b636-3c53080e34c7" containerName="kserve-container"
Jul 01 17:10:16.545924 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:16.545906    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ce016-predictor-6ffdb68b79-hq49w"
Jul 01 17:10:16.548742 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:16.548720    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-ce016-predictor-serving-cert\""
Jul 01 17:10:16.548823 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:16.548750    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-ce016-kube-rbac-proxy-sar-config\""
Jul 01 17:10:16.560288 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:16.560259    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ce016-predictor-6ffdb68b79-hq49w"]
Jul 01 17:10:16.584932 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:16.584893    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:10:16.681647 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:16.681611    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bee5b713-8b32-437d-86ad-c6c646b805c3-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-ce016-predictor-6ffdb68b79-hq49w\" (UID: \"bee5b713-8b32-437d-86ad-c6c646b805c3\") " pod="kserve-ci-e2e-test/success-200-isvc-ce016-predictor-6ffdb68b79-hq49w"
Jul 01 17:10:16.681823 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:16.681652    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9t2c\" (UniqueName: \"kubernetes.io/projected/bee5b713-8b32-437d-86ad-c6c646b805c3-kube-api-access-r9t2c\") pod \"success-200-isvc-ce016-predictor-6ffdb68b79-hq49w\" (UID: \"bee5b713-8b32-437d-86ad-c6c646b805c3\") " pod="kserve-ci-e2e-test/success-200-isvc-ce016-predictor-6ffdb68b79-hq49w"
Jul 01 17:10:16.681823 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:16.681680    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bee5b713-8b32-437d-86ad-c6c646b805c3-proxy-tls\") pod \"success-200-isvc-ce016-predictor-6ffdb68b79-hq49w\" (UID: \"bee5b713-8b32-437d-86ad-c6c646b805c3\") " pod="kserve-ci-e2e-test/success-200-isvc-ce016-predictor-6ffdb68b79-hq49w"
Jul 01 17:10:16.684739 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:16.684699    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:10:16.688171 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:16.688145    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ce016-predictor-84dbff7cdb-rdb8n"]
Jul 01 17:10:16.692665 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:16.692644    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-ce016-predictor-84dbff7cdb-rdb8n"
Jul 01 17:10:16.696106 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:16.696086    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-ce016-predictor-serving-cert\""
Jul 01 17:10:16.696573 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:16.696551    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-ce016-kube-rbac-proxy-sar-config\""
Jul 01 17:10:16.712021 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:16.711988    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ce016-predictor-84dbff7cdb-rdb8n"]
Jul 01 17:10:16.782817 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:16.782774    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bee5b713-8b32-437d-86ad-c6c646b805c3-proxy-tls\") pod \"success-200-isvc-ce016-predictor-6ffdb68b79-hq49w\" (UID: \"bee5b713-8b32-437d-86ad-c6c646b805c3\") " pod="kserve-ci-e2e-test/success-200-isvc-ce016-predictor-6ffdb68b79-hq49w"
Jul 01 17:10:16.783009 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:16.782852    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/318c4e47-af61-4745-be04-a6ca2637ed1e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-ce016-predictor-84dbff7cdb-rdb8n\" (UID: \"318c4e47-af61-4745-be04-a6ca2637ed1e\") " pod="kserve-ci-e2e-test/error-404-isvc-ce016-predictor-84dbff7cdb-rdb8n"
Jul 01 17:10:16.783009 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:16.782910    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkxnz\" (UniqueName: \"kubernetes.io/projected/318c4e47-af61-4745-be04-a6ca2637ed1e-kube-api-access-bkxnz\") pod \"error-404-isvc-ce016-predictor-84dbff7cdb-rdb8n\" (UID: \"318c4e47-af61-4745-be04-a6ca2637ed1e\") " pod="kserve-ci-e2e-test/error-404-isvc-ce016-predictor-84dbff7cdb-rdb8n"
Jul 01 17:10:16.783009 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:10:16.782946    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-ce016-predictor-serving-cert: secret "success-200-isvc-ce016-predictor-serving-cert" not found
Jul 01 17:10:16.783009 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:16.782981    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/318c4e47-af61-4745-be04-a6ca2637ed1e-proxy-tls\") pod \"error-404-isvc-ce016-predictor-84dbff7cdb-rdb8n\" (UID: \"318c4e47-af61-4745-be04-a6ca2637ed1e\") " pod="kserve-ci-e2e-test/error-404-isvc-ce016-predictor-84dbff7cdb-rdb8n"
Jul 01 17:10:16.783231 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:16.783015    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bee5b713-8b32-437d-86ad-c6c646b805c3-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-ce016-predictor-6ffdb68b79-hq49w\" (UID: \"bee5b713-8b32-437d-86ad-c6c646b805c3\") " pod="kserve-ci-e2e-test/success-200-isvc-ce016-predictor-6ffdb68b79-hq49w"
Jul 01 17:10:16.783231 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:10:16.783037    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bee5b713-8b32-437d-86ad-c6c646b805c3-proxy-tls podName:bee5b713-8b32-437d-86ad-c6c646b805c3 nodeName:}" failed. No retries permitted until 2026-07-01 17:10:17.28301413 +0000 UTC m=+735.080152524 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/bee5b713-8b32-437d-86ad-c6c646b805c3-proxy-tls") pod "success-200-isvc-ce016-predictor-6ffdb68b79-hq49w" (UID: "bee5b713-8b32-437d-86ad-c6c646b805c3") : secret "success-200-isvc-ce016-predictor-serving-cert" not found
Jul 01 17:10:16.783231 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:16.783071    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9t2c\" (UniqueName: \"kubernetes.io/projected/bee5b713-8b32-437d-86ad-c6c646b805c3-kube-api-access-r9t2c\") pod \"success-200-isvc-ce016-predictor-6ffdb68b79-hq49w\" (UID: \"bee5b713-8b32-437d-86ad-c6c646b805c3\") " pod="kserve-ci-e2e-test/success-200-isvc-ce016-predictor-6ffdb68b79-hq49w"
Jul 01 17:10:16.783705 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:16.783683    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bee5b713-8b32-437d-86ad-c6c646b805c3-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-ce016-predictor-6ffdb68b79-hq49w\" (UID: \"bee5b713-8b32-437d-86ad-c6c646b805c3\") " pod="kserve-ci-e2e-test/success-200-isvc-ce016-predictor-6ffdb68b79-hq49w"
Jul 01 17:10:16.795436 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:16.795409    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9t2c\" (UniqueName: \"kubernetes.io/projected/bee5b713-8b32-437d-86ad-c6c646b805c3-kube-api-access-r9t2c\") pod \"success-200-isvc-ce016-predictor-6ffdb68b79-hq49w\" (UID: \"bee5b713-8b32-437d-86ad-c6c646b805c3\") " pod="kserve-ci-e2e-test/success-200-isvc-ce016-predictor-6ffdb68b79-hq49w"
Jul 01 17:10:16.850499 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:16.850407    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:10:16.883689 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:16.883662    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/318c4e47-af61-4745-be04-a6ca2637ed1e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-ce016-predictor-84dbff7cdb-rdb8n\" (UID: \"318c4e47-af61-4745-be04-a6ca2637ed1e\") " pod="kserve-ci-e2e-test/error-404-isvc-ce016-predictor-84dbff7cdb-rdb8n"
Jul 01 17:10:16.883860 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:16.883709    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkxnz\" (UniqueName: \"kubernetes.io/projected/318c4e47-af61-4745-be04-a6ca2637ed1e-kube-api-access-bkxnz\") pod \"error-404-isvc-ce016-predictor-84dbff7cdb-rdb8n\" (UID: \"318c4e47-af61-4745-be04-a6ca2637ed1e\") " pod="kserve-ci-e2e-test/error-404-isvc-ce016-predictor-84dbff7cdb-rdb8n"
Jul 01 17:10:16.883860 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:16.883755    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/318c4e47-af61-4745-be04-a6ca2637ed1e-proxy-tls\") pod \"error-404-isvc-ce016-predictor-84dbff7cdb-rdb8n\" (UID: \"318c4e47-af61-4745-be04-a6ca2637ed1e\") " pod="kserve-ci-e2e-test/error-404-isvc-ce016-predictor-84dbff7cdb-rdb8n"
Jul 01 17:10:16.884315 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:16.884298    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/318c4e47-af61-4745-be04-a6ca2637ed1e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-ce016-predictor-84dbff7cdb-rdb8n\" (UID: \"318c4e47-af61-4745-be04-a6ca2637ed1e\") " pod="kserve-ci-e2e-test/error-404-isvc-ce016-predictor-84dbff7cdb-rdb8n"
Jul 01 17:10:16.886307 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:16.886287    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/318c4e47-af61-4745-be04-a6ca2637ed1e-proxy-tls\") pod \"error-404-isvc-ce016-predictor-84dbff7cdb-rdb8n\" (UID: \"318c4e47-af61-4745-be04-a6ca2637ed1e\") " pod="kserve-ci-e2e-test/error-404-isvc-ce016-predictor-84dbff7cdb-rdb8n"
Jul 01 17:10:16.898289 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:16.898269    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkxnz\" (UniqueName: \"kubernetes.io/projected/318c4e47-af61-4745-be04-a6ca2637ed1e-kube-api-access-bkxnz\") pod \"error-404-isvc-ce016-predictor-84dbff7cdb-rdb8n\" (UID: \"318c4e47-af61-4745-be04-a6ca2637ed1e\") " pod="kserve-ci-e2e-test/error-404-isvc-ce016-predictor-84dbff7cdb-rdb8n"
Jul 01 17:10:16.951918 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:16.951876    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:10:17.005963 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:17.005932    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-ce016-predictor-84dbff7cdb-rdb8n"
Jul 01 17:10:17.160447 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:17.160365    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ce016-predictor-84dbff7cdb-rdb8n"]
Jul 01 17:10:17.163185 ip-10-0-137-180 kubenswrapper[2563]: W0701 17:10:17.163151    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod318c4e47_af61_4745_be04_a6ca2637ed1e.slice/crio-a9695844ccc1a8fd8098ac5eaa7c066019c685d25471e98e74defa80e0cbc5bc WatchSource:0}: Error finding container a9695844ccc1a8fd8098ac5eaa7c066019c685d25471e98e74defa80e0cbc5bc: Status 404 returned error can't find the container with id a9695844ccc1a8fd8098ac5eaa7c066019c685d25471e98e74defa80e0cbc5bc
Jul 01 17:10:17.287359 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:17.287318    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bee5b713-8b32-437d-86ad-c6c646b805c3-proxy-tls\") pod \"success-200-isvc-ce016-predictor-6ffdb68b79-hq49w\" (UID: \"bee5b713-8b32-437d-86ad-c6c646b805c3\") " pod="kserve-ci-e2e-test/success-200-isvc-ce016-predictor-6ffdb68b79-hq49w"
Jul 01 17:10:17.289635 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:17.289616    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bee5b713-8b32-437d-86ad-c6c646b805c3-proxy-tls\") pod \"success-200-isvc-ce016-predictor-6ffdb68b79-hq49w\" (UID: \"bee5b713-8b32-437d-86ad-c6c646b805c3\") " pod="kserve-ci-e2e-test/success-200-isvc-ce016-predictor-6ffdb68b79-hq49w"
Jul 01 17:10:17.442189 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:17.442098    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3c336-predictor-54665d447b-s2vc2" podUID="0ce82ad2-7236-475b-b3ff-eccfa49eccfa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 01 17:10:17.457323 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:17.457292    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ce016-predictor-6ffdb68b79-hq49w"
Jul 01 17:10:17.552336 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:17.552302    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ce016-predictor-84dbff7cdb-rdb8n" event={"ID":"318c4e47-af61-4745-be04-a6ca2637ed1e","Type":"ContainerStarted","Data":"ccdf69b4fd13e204738793a3890ee4b0ed6e8d50d5334d772710f529dab71379"}
Jul 01 17:10:17.552532 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:17.552385    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ce016-predictor-84dbff7cdb-rdb8n" event={"ID":"318c4e47-af61-4745-be04-a6ca2637ed1e","Type":"ContainerStarted","Data":"57712d8baa361854be1ce1ba6abc303567ea7897a4316e2078a49c767c3b38ef"}
Jul 01 17:10:17.552532 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:17.552417    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ce016-predictor-84dbff7cdb-rdb8n" event={"ID":"318c4e47-af61-4745-be04-a6ca2637ed1e","Type":"ContainerStarted","Data":"a9695844ccc1a8fd8098ac5eaa7c066019c685d25471e98e74defa80e0cbc5bc"}
Jul 01 17:10:17.552654 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:17.552545    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-ce016-predictor-84dbff7cdb-rdb8n"
Jul 01 17:10:17.573381 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:17.573307    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-ce016-predictor-84dbff7cdb-rdb8n" podStartSLOduration=1.573287026 podStartE2EDuration="1.573287026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:10:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:10:17.571529568 +0000 UTC m=+735.368667984" watchObservedRunningTime="2026-07-01 17:10:17.573287026 +0000 UTC m=+735.370425429"
Jul 01 17:10:17.593519 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:17.593497    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ce016-predictor-6ffdb68b79-hq49w"]
Jul 01 17:10:17.595874 ip-10-0-137-180 kubenswrapper[2563]: W0701 17:10:17.595846    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbee5b713_8b32_437d_86ad_c6c646b805c3.slice/crio-a01535f9af0e3b38a865602643438b91b609ffef0c514f96db7dbe39ecc227d8 WatchSource:0}: Error finding container a01535f9af0e3b38a865602643438b91b609ffef0c514f96db7dbe39ecc227d8: Status 404 returned error can't find the container with id a01535f9af0e3b38a865602643438b91b609ffef0c514f96db7dbe39ecc227d8
Jul 01 17:10:18.448244 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:18.448199    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3c336-predictor-547bb6bc8f-96z6b" podUID="dc760277-8744-4acb-89dc-4cbc32f83742" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 01 17:10:18.557927 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:18.557889    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ce016-predictor-6ffdb68b79-hq49w" event={"ID":"bee5b713-8b32-437d-86ad-c6c646b805c3","Type":"ContainerStarted","Data":"c391b181f049b8884f6b34bdd1c754b113af19c280b9a33df7d1552395692064"}
Jul 01 17:10:18.558104 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:18.557957    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ce016-predictor-6ffdb68b79-hq49w" event={"ID":"bee5b713-8b32-437d-86ad-c6c646b805c3","Type":"ContainerStarted","Data":"8973fa6bb27fafb35d84a407e0c752d080de1f6fb7fe9fb4c5e7804030c8a1d9"}
Jul 01 17:10:18.558104 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:18.557989    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ce016-predictor-6ffdb68b79-hq49w" event={"ID":"bee5b713-8b32-437d-86ad-c6c646b805c3","Type":"ContainerStarted","Data":"a01535f9af0e3b38a865602643438b91b609ffef0c514f96db7dbe39ecc227d8"}
Jul 01 17:10:18.558191 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:18.558116    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-ce016-predictor-84dbff7cdb-rdb8n"
Jul 01 17:10:18.559618 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:18.559594    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ce016-predictor-84dbff7cdb-rdb8n" podUID="318c4e47-af61-4745-be04-a6ca2637ed1e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 01 17:10:18.577813 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:18.577766    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-ce016-predictor-6ffdb68b79-hq49w" podStartSLOduration=2.577751324 podStartE2EDuration="2.577751324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:10:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:10:18.577110021 +0000 UTC m=+736.374248444" watchObservedRunningTime="2026-07-01 17:10:18.577751324 +0000 UTC m=+736.374889725"
Jul 01 17:10:19.562049 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:19.562008    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ce016-predictor-84dbff7cdb-rdb8n" podUID="318c4e47-af61-4745-be04-a6ca2637ed1e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 01 17:10:19.562486 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:19.562158    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-ce016-predictor-6ffdb68b79-hq49w"
Jul 01 17:10:19.562486 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:19.562206    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-ce016-predictor-6ffdb68b79-hq49w"
Jul 01 17:10:19.563650 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:19.563582    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ce016-predictor-6ffdb68b79-hq49w" podUID="bee5b713-8b32-437d-86ad-c6c646b805c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Jul 01 17:10:20.545319 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:20.545295    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-1b15a-5bf689b779-zll7z"
Jul 01 17:10:20.567527 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:20.567494    2563 generic.go:356] "Generic (PLEG): container finished" podID="d52c91d6-3f99-4d01-9226-ce261eb999b1" containerID="9b902510d03a129bb17edd823ca85c9a4baa216184441ddafc9172e4ac636187" exitCode=137
Jul 01 17:10:20.567879 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:20.567534    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-1b15a-5bf689b779-zll7z"
Jul 01 17:10:20.567879 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:20.567542    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-1b15a-5bf689b779-zll7z" event={"ID":"d52c91d6-3f99-4d01-9226-ce261eb999b1","Type":"ContainerDied","Data":"9b902510d03a129bb17edd823ca85c9a4baa216184441ddafc9172e4ac636187"}
Jul 01 17:10:20.567879 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:20.567637    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-1b15a-5bf689b779-zll7z" event={"ID":"d52c91d6-3f99-4d01-9226-ce261eb999b1","Type":"ContainerDied","Data":"0a3c18c00fb80e5c710440c906f5af61d918134a471c804bccb2c214bf54b758"}
Jul 01 17:10:20.567879 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:20.567670    2563 scope.go:117] "RemoveContainer" containerID="9b902510d03a129bb17edd823ca85c9a4baa216184441ddafc9172e4ac636187"
Jul 01 17:10:20.568080 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:20.568057    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ce016-predictor-6ffdb68b79-hq49w" podUID="bee5b713-8b32-437d-86ad-c6c646b805c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Jul 01 17:10:20.582669 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:20.582650    2563 scope.go:117] "RemoveContainer" containerID="9b902510d03a129bb17edd823ca85c9a4baa216184441ddafc9172e4ac636187"
Jul 01 17:10:20.582936 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:10:20.582919    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b902510d03a129bb17edd823ca85c9a4baa216184441ddafc9172e4ac636187\": container with ID starting with 9b902510d03a129bb17edd823ca85c9a4baa216184441ddafc9172e4ac636187 not found: ID does not exist" containerID="9b902510d03a129bb17edd823ca85c9a4baa216184441ddafc9172e4ac636187"
Jul 01 17:10:20.582992 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:20.582946    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b902510d03a129bb17edd823ca85c9a4baa216184441ddafc9172e4ac636187"} err="failed to get container status \"9b902510d03a129bb17edd823ca85c9a4baa216184441ddafc9172e4ac636187\": rpc error: code = NotFound desc = could not find container \"9b902510d03a129bb17edd823ca85c9a4baa216184441ddafc9172e4ac636187\": container with ID starting with 9b902510d03a129bb17edd823ca85c9a4baa216184441ddafc9172e4ac636187 not found: ID does not exist"
Jul 01 17:10:20.617482 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:20.617445    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d52c91d6-3f99-4d01-9226-ce261eb999b1-proxy-tls\") pod \"d52c91d6-3f99-4d01-9226-ce261eb999b1\" (UID: \"d52c91d6-3f99-4d01-9226-ce261eb999b1\") "
Jul 01 17:10:20.617616 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:20.617547    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d52c91d6-3f99-4d01-9226-ce261eb999b1-openshift-service-ca-bundle\") pod \"d52c91d6-3f99-4d01-9226-ce261eb999b1\" (UID: \"d52c91d6-3f99-4d01-9226-ce261eb999b1\") "
Jul 01 17:10:20.618060 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:20.618024    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d52c91d6-3f99-4d01-9226-ce261eb999b1-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "d52c91d6-3f99-4d01-9226-ce261eb999b1" (UID: "d52c91d6-3f99-4d01-9226-ce261eb999b1"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:10:20.619668 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:20.619646    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d52c91d6-3f99-4d01-9226-ce261eb999b1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d52c91d6-3f99-4d01-9226-ce261eb999b1" (UID: "d52c91d6-3f99-4d01-9226-ce261eb999b1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 17:10:20.718782 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:20.718753    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d52c91d6-3f99-4d01-9226-ce261eb999b1-openshift-service-ca-bundle\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:10:20.718782 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:20.718783    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d52c91d6-3f99-4d01-9226-ce261eb999b1-proxy-tls\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:10:20.884240 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:20.884208    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-1b15a-5bf689b779-zll7z"]
Jul 01 17:10:20.887938 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:20.887914    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-1b15a-5bf689b779-zll7z"]
Jul 01 17:10:22.696871 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:22.696835    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d52c91d6-3f99-4d01-9226-ce261eb999b1" path="/var/lib/kubelet/pods/d52c91d6-3f99-4d01-9226-ce261eb999b1/volumes"
Jul 01 17:10:24.567009 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:24.566978    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-ce016-predictor-84dbff7cdb-rdb8n"
Jul 01 17:10:24.567460 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:24.567432    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ce016-predictor-84dbff7cdb-rdb8n" podUID="318c4e47-af61-4745-be04-a6ca2637ed1e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 01 17:10:25.572268 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:25.572242    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-ce016-predictor-6ffdb68b79-hq49w"
Jul 01 17:10:25.572714 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:25.572677    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ce016-predictor-6ffdb68b79-hq49w" podUID="bee5b713-8b32-437d-86ad-c6c646b805c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Jul 01 17:10:27.441675 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:27.441630    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3c336-predictor-54665d447b-s2vc2" podUID="0ce82ad2-7236-475b-b3ff-eccfa49eccfa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 01 17:10:28.448051 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:28.448008    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3c336-predictor-547bb6bc8f-96z6b" podUID="dc760277-8744-4acb-89dc-4cbc32f83742" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 01 17:10:29.665123 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:29.665080    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:10:29.727318 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:29.727280    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:10:34.567431 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:34.567383    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ce016-predictor-84dbff7cdb-rdb8n" podUID="318c4e47-af61-4745-be04-a6ca2637ed1e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 01 17:10:35.573617 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:35.573576    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ce016-predictor-6ffdb68b79-hq49w" podUID="bee5b713-8b32-437d-86ad-c6c646b805c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Jul 01 17:10:37.442257 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:37.442226    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-3c336-predictor-54665d447b-s2vc2"
Jul 01 17:10:38.447828 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:38.447790    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3c336-predictor-547bb6bc8f-96z6b" podUID="dc760277-8744-4acb-89dc-4cbc32f83742" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 01 17:10:44.567742 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:44.567701    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ce016-predictor-84dbff7cdb-rdb8n" podUID="318c4e47-af61-4745-be04-a6ca2637ed1e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 01 17:10:45.573542 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:45.573498    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ce016-predictor-6ffdb68b79-hq49w" podUID="bee5b713-8b32-437d-86ad-c6c646b805c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Jul 01 17:10:46.049156 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:46.049116    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:10:46.122899 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:46.122863    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:10:48.448295 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:48.448262    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-3c336-predictor-547bb6bc8f-96z6b"
Jul 01 17:10:54.568107 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:54.568069    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ce016-predictor-84dbff7cdb-rdb8n" podUID="318c4e47-af61-4745-be04-a6ca2637ed1e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 01 17:10:55.572735 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:10:55.572698    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ce016-predictor-6ffdb68b79-hq49w" podUID="bee5b713-8b32-437d-86ad-c6c646b805c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Jul 01 17:11:00.630251 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:11:00.630212    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:11:00.993726 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:11:00.993687    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:11:04.569132 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:11:04.569093    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-ce016-predictor-84dbff7cdb-rdb8n"
Jul 01 17:11:05.574528 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:11:05.574500    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-ce016-predictor-6ffdb68b79-hq49w"
Jul 01 17:11:16.471968 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:11:16.471925    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:11:16.571649 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:11:16.571615    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:12:35.932206 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:12:35.932164    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:12:36.031576 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:12:36.031503    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:14:24.387240 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:14:24.387152    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:16:41.514665 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:16:41.514631    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:16:43.381543 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:16:43.381508    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:17:43.770614 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:17:43.770580    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:19:15.376893 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:15.376809    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:19:15.480142 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:15.480112    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:19:15.525526 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:15.525494    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-52e43-predictor-674cd6cc9-prtkw"]
Jul 01 17:19:15.525842 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:15.525829    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d52c91d6-3f99-4d01-9226-ce261eb999b1" containerName="switch-graph-1b15a"
Jul 01 17:19:15.525910 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:15.525845    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d52c91d6-3f99-4d01-9226-ce261eb999b1" containerName="switch-graph-1b15a"
Jul 01 17:19:15.525910 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:15.525909    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d52c91d6-3f99-4d01-9226-ce261eb999b1" containerName="switch-graph-1b15a"
Jul 01 17:19:15.529065 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:15.529042    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-52e43-predictor-674cd6cc9-prtkw"
Jul 01 17:19:15.529196 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:15.529071    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3c336-predictor-547bb6bc8f-96z6b"]
Jul 01 17:19:15.529501 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:15.529379    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-3c336-predictor-547bb6bc8f-96z6b" podUID="dc760277-8744-4acb-89dc-4cbc32f83742" containerName="kube-rbac-proxy" containerID="cri-o://c0d752993c86d6c7a8d349f2444a6d7030def616cbc34f7cb85a847cd7ada222" gracePeriod=30
Jul 01 17:19:15.529501 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:15.529438    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-3c336-predictor-547bb6bc8f-96z6b" podUID="dc760277-8744-4acb-89dc-4cbc32f83742" containerName="kserve-container" containerID="cri-o://9f335543f9b6ea288b41e81269025946946c16bb316bad568b859148f62e1686" gracePeriod=30
Jul 01 17:19:15.531851 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:15.531826    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-52e43-predictor-serving-cert\""
Jul 01 17:19:15.531992 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:15.531863    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-52e43-kube-rbac-proxy-sar-config\""
Jul 01 17:19:15.544234 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:15.544210    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-52e43-predictor-674cd6cc9-prtkw"]
Jul 01 17:19:15.584192 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:15.584152    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:19:15.589659 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:15.589632    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/430b61ee-095e-470f-a487-7d790277b37d-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-52e43-predictor-674cd6cc9-prtkw\" (UID: \"430b61ee-095e-470f-a487-7d790277b37d\") " pod="kserve-ci-e2e-test/success-200-isvc-52e43-predictor-674cd6cc9-prtkw"
Jul 01 17:19:15.589813 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:15.589681    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/430b61ee-095e-470f-a487-7d790277b37d-proxy-tls\") pod \"success-200-isvc-52e43-predictor-674cd6cc9-prtkw\" (UID: \"430b61ee-095e-470f-a487-7d790277b37d\") " pod="kserve-ci-e2e-test/success-200-isvc-52e43-predictor-674cd6cc9-prtkw"
Jul 01 17:19:15.589813 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:15.589708    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8spwr\" (UniqueName: \"kubernetes.io/projected/430b61ee-095e-470f-a487-7d790277b37d-kube-api-access-8spwr\") pod \"success-200-isvc-52e43-predictor-674cd6cc9-prtkw\" (UID: \"430b61ee-095e-470f-a487-7d790277b37d\") " pod="kserve-ci-e2e-test/success-200-isvc-52e43-predictor-674cd6cc9-prtkw"
Jul 01 17:19:15.593027 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:19:15.592990    2563 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc760277_8744_4acb_89dc_4cbc32f83742.slice/crio-conmon-c0d752993c86d6c7a8d349f2444a6d7030def616cbc34f7cb85a847cd7ada222.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc760277_8744_4acb_89dc_4cbc32f83742.slice/crio-c0d752993c86d6c7a8d349f2444a6d7030def616cbc34f7cb85a847cd7ada222.scope\": RecentStats: unable to find data in memory cache]"
Jul 01 17:19:15.602988 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:15.602961    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-52e43-predictor-79768fc67d-xk27k"]
Jul 01 17:19:15.606413 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:15.606386    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-52e43-predictor-79768fc67d-xk27k"
Jul 01 17:19:15.609381 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:15.609334    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-52e43-kube-rbac-proxy-sar-config\""
Jul 01 17:19:15.609471 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:15.609379    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-52e43-predictor-serving-cert\""
Jul 01 17:19:15.618706 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:15.618685    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-52e43-predictor-79768fc67d-xk27k"]
Jul 01 17:19:15.663301 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:15.663271    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3c336-predictor-54665d447b-s2vc2"]
Jul 01 17:19:15.663568 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:15.663544    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-3c336-predictor-54665d447b-s2vc2" podUID="0ce82ad2-7236-475b-b3ff-eccfa49eccfa" containerName="kserve-container" containerID="cri-o://624b66f1b669b1173df86310452a5c315ded3f96a559c446c912fd16e2a1723b" gracePeriod=30
Jul 01 17:19:15.663692 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:15.663646    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-3c336-predictor-54665d447b-s2vc2" podUID="0ce82ad2-7236-475b-b3ff-eccfa49eccfa" containerName="kube-rbac-proxy" containerID="cri-o://0b7ad5277b8472209ebfc568435438f7aa1fb5739fc8d1f25a128c2a020f59b1" gracePeriod=30
Jul 01 17:19:15.676007 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:15.675980    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:19:15.690132 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:15.690106    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/430b61ee-095e-470f-a487-7d790277b37d-proxy-tls\") pod \"success-200-isvc-52e43-predictor-674cd6cc9-prtkw\" (UID: \"430b61ee-095e-470f-a487-7d790277b37d\") " pod="kserve-ci-e2e-test/success-200-isvc-52e43-predictor-674cd6cc9-prtkw"
Jul 01 17:19:15.690323 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:15.690138    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8spwr\" (UniqueName: \"kubernetes.io/projected/430b61ee-095e-470f-a487-7d790277b37d-kube-api-access-8spwr\") pod \"success-200-isvc-52e43-predictor-674cd6cc9-prtkw\" (UID: \"430b61ee-095e-470f-a487-7d790277b37d\") " pod="kserve-ci-e2e-test/success-200-isvc-52e43-predictor-674cd6cc9-prtkw"
Jul 01 17:19:15.690323 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:15.690169    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e49336bc-9319-467b-a6b8-969be03815e0-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-52e43-predictor-79768fc67d-xk27k\" (UID: \"e49336bc-9319-467b-a6b8-969be03815e0\") " pod="kserve-ci-e2e-test/error-404-isvc-52e43-predictor-79768fc67d-xk27k"
Jul 01 17:19:15.690323 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:15.690218    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e49336bc-9319-467b-a6b8-969be03815e0-proxy-tls\") pod \"error-404-isvc-52e43-predictor-79768fc67d-xk27k\" (UID: \"e49336bc-9319-467b-a6b8-969be03815e0\") " pod="kserve-ci-e2e-test/error-404-isvc-52e43-predictor-79768fc67d-xk27k"
Jul 01 17:19:15.690323 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:15.690267    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2ctl\" (UniqueName: \"kubernetes.io/projected/e49336bc-9319-467b-a6b8-969be03815e0-kube-api-access-s2ctl\") pod \"error-404-isvc-52e43-predictor-79768fc67d-xk27k\" (UID: \"e49336bc-9319-467b-a6b8-969be03815e0\") " pod="kserve-ci-e2e-test/error-404-isvc-52e43-predictor-79768fc67d-xk27k"
Jul 01 17:19:15.690323 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:19:15.690300    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-52e43-predictor-serving-cert: secret "success-200-isvc-52e43-predictor-serving-cert" not found
Jul 01 17:19:15.690677 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:19:15.690398    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/430b61ee-095e-470f-a487-7d790277b37d-proxy-tls podName:430b61ee-095e-470f-a487-7d790277b37d nodeName:}" failed. No retries permitted until 2026-07-01 17:19:16.19037451 +0000 UTC m=+1273.987512895 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/430b61ee-095e-470f-a487-7d790277b37d-proxy-tls") pod "success-200-isvc-52e43-predictor-674cd6cc9-prtkw" (UID: "430b61ee-095e-470f-a487-7d790277b37d") : secret "success-200-isvc-52e43-predictor-serving-cert" not found
Jul 01 17:19:15.690677 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:15.690441    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/430b61ee-095e-470f-a487-7d790277b37d-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-52e43-predictor-674cd6cc9-prtkw\" (UID: \"430b61ee-095e-470f-a487-7d790277b37d\") " pod="kserve-ci-e2e-test/success-200-isvc-52e43-predictor-674cd6cc9-prtkw"
Jul 01 17:19:15.691150 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:15.691118    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/430b61ee-095e-470f-a487-7d790277b37d-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-52e43-predictor-674cd6cc9-prtkw\" (UID: \"430b61ee-095e-470f-a487-7d790277b37d\") " pod="kserve-ci-e2e-test/success-200-isvc-52e43-predictor-674cd6cc9-prtkw"
Jul 01 17:19:15.694530 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:15.694505    2563 generic.go:356] "Generic (PLEG): container finished" podID="dc760277-8744-4acb-89dc-4cbc32f83742" containerID="c0d752993c86d6c7a8d349f2444a6d7030def616cbc34f7cb85a847cd7ada222" exitCode=2
Jul 01 17:19:15.694640 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:15.694588    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3c336-predictor-547bb6bc8f-96z6b" event={"ID":"dc760277-8744-4acb-89dc-4cbc32f83742","Type":"ContainerDied","Data":"c0d752993c86d6c7a8d349f2444a6d7030def616cbc34f7cb85a847cd7ada222"}
Jul 01 17:19:15.699200 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:15.699177    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8spwr\" (UniqueName: \"kubernetes.io/projected/430b61ee-095e-470f-a487-7d790277b37d-kube-api-access-8spwr\") pod \"success-200-isvc-52e43-predictor-674cd6cc9-prtkw\" (UID: \"430b61ee-095e-470f-a487-7d790277b37d\") " pod="kserve-ci-e2e-test/success-200-isvc-52e43-predictor-674cd6cc9-prtkw"
Jul 01 17:19:15.775115 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:15.775082    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:19:15.791391 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:15.791337    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e49336bc-9319-467b-a6b8-969be03815e0-proxy-tls\") pod \"error-404-isvc-52e43-predictor-79768fc67d-xk27k\" (UID: \"e49336bc-9319-467b-a6b8-969be03815e0\") " pod="kserve-ci-e2e-test/error-404-isvc-52e43-predictor-79768fc67d-xk27k"
Jul 01 17:19:15.791498 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:15.791443    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2ctl\" (UniqueName: \"kubernetes.io/projected/e49336bc-9319-467b-a6b8-969be03815e0-kube-api-access-s2ctl\") pod \"error-404-isvc-52e43-predictor-79768fc67d-xk27k\" (UID: \"e49336bc-9319-467b-a6b8-969be03815e0\") " pod="kserve-ci-e2e-test/error-404-isvc-52e43-predictor-79768fc67d-xk27k"
Jul 01 17:19:15.791588 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:15.791572    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e49336bc-9319-467b-a6b8-969be03815e0-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-52e43-predictor-79768fc67d-xk27k\" (UID: \"e49336bc-9319-467b-a6b8-969be03815e0\") " pod="kserve-ci-e2e-test/error-404-isvc-52e43-predictor-79768fc67d-xk27k"
Jul 01 17:19:15.792190 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:15.792127    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e49336bc-9319-467b-a6b8-969be03815e0-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-52e43-predictor-79768fc67d-xk27k\" (UID: \"e49336bc-9319-467b-a6b8-969be03815e0\") " pod="kserve-ci-e2e-test/error-404-isvc-52e43-predictor-79768fc67d-xk27k"
Jul 01 17:19:15.793825 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:15.793804    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e49336bc-9319-467b-a6b8-969be03815e0-proxy-tls\") pod \"error-404-isvc-52e43-predictor-79768fc67d-xk27k\" (UID: \"e49336bc-9319-467b-a6b8-969be03815e0\") " pod="kserve-ci-e2e-test/error-404-isvc-52e43-predictor-79768fc67d-xk27k"
Jul 01 17:19:15.800415 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:15.800396    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2ctl\" (UniqueName: \"kubernetes.io/projected/e49336bc-9319-467b-a6b8-969be03815e0-kube-api-access-s2ctl\") pod \"error-404-isvc-52e43-predictor-79768fc67d-xk27k\" (UID: \"e49336bc-9319-467b-a6b8-969be03815e0\") " pod="kserve-ci-e2e-test/error-404-isvc-52e43-predictor-79768fc67d-xk27k"
Jul 01 17:19:15.916550 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:15.916453    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-52e43-predictor-79768fc67d-xk27k"
Jul 01 17:19:16.053247 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:16.053155    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-52e43-predictor-79768fc67d-xk27k"]
Jul 01 17:19:16.056919 ip-10-0-137-180 kubenswrapper[2563]: W0701 17:19:16.056894    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode49336bc_9319_467b_a6b8_969be03815e0.slice/crio-12b5a2ab49208f7229b2ff7e0b66688f1586146fa1ac8305abfe97536e515cf0 WatchSource:0}: Error finding container 12b5a2ab49208f7229b2ff7e0b66688f1586146fa1ac8305abfe97536e515cf0: Status 404 returned error can't find the container with id 12b5a2ab49208f7229b2ff7e0b66688f1586146fa1ac8305abfe97536e515cf0
Jul 01 17:19:16.142940 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:16.142908    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:19:16.195020 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:16.194990    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/430b61ee-095e-470f-a487-7d790277b37d-proxy-tls\") pod \"success-200-isvc-52e43-predictor-674cd6cc9-prtkw\" (UID: \"430b61ee-095e-470f-a487-7d790277b37d\") " pod="kserve-ci-e2e-test/success-200-isvc-52e43-predictor-674cd6cc9-prtkw"
Jul 01 17:19:16.197424 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:16.197395    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/430b61ee-095e-470f-a487-7d790277b37d-proxy-tls\") pod \"success-200-isvc-52e43-predictor-674cd6cc9-prtkw\" (UID: \"430b61ee-095e-470f-a487-7d790277b37d\") " pod="kserve-ci-e2e-test/success-200-isvc-52e43-predictor-674cd6cc9-prtkw"
Jul 01 17:19:16.453620 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:16.453533    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-52e43-predictor-674cd6cc9-prtkw"
Jul 01 17:19:16.586324 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:16.586300    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-52e43-predictor-674cd6cc9-prtkw"]
Jul 01 17:19:16.588414 ip-10-0-137-180 kubenswrapper[2563]: W0701 17:19:16.588377    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod430b61ee_095e_470f_a487_7d790277b37d.slice/crio-29703f53293de1a8ce91cec427afcda1e102f2a4eb75e24aa33249e583a500de WatchSource:0}: Error finding container 29703f53293de1a8ce91cec427afcda1e102f2a4eb75e24aa33249e583a500de: Status 404 returned error can't find the container with id 29703f53293de1a8ce91cec427afcda1e102f2a4eb75e24aa33249e583a500de
Jul 01 17:19:16.703884 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:16.703811    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-52e43-predictor-79768fc67d-xk27k" event={"ID":"e49336bc-9319-467b-a6b8-969be03815e0","Type":"ContainerStarted","Data":"b67ca1549736c9d0c079978edcbf7e352faae97b5d7de801fb52049174d900b9"}
Jul 01 17:19:16.703884 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:16.703879    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-52e43-predictor-79768fc67d-xk27k" event={"ID":"e49336bc-9319-467b-a6b8-969be03815e0","Type":"ContainerStarted","Data":"e4e01e2a3a4246ebfe8d9a9aa6defbdddb1049c0553b307828c3677bd661ddc0"}
Jul 01 17:19:16.704075 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:16.703911    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-52e43-predictor-79768fc67d-xk27k" event={"ID":"e49336bc-9319-467b-a6b8-969be03815e0","Type":"ContainerStarted","Data":"12b5a2ab49208f7229b2ff7e0b66688f1586146fa1ac8305abfe97536e515cf0"}
Jul 01 17:19:16.704075 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:16.703945    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-52e43-predictor-79768fc67d-xk27k"
Jul 01 17:19:16.705502 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:16.705468    2563 generic.go:356] "Generic (PLEG): container finished" podID="0ce82ad2-7236-475b-b3ff-eccfa49eccfa" containerID="0b7ad5277b8472209ebfc568435438f7aa1fb5739fc8d1f25a128c2a020f59b1" exitCode=2
Jul 01 17:19:16.705616 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:16.705536    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3c336-predictor-54665d447b-s2vc2" event={"ID":"0ce82ad2-7236-475b-b3ff-eccfa49eccfa","Type":"ContainerDied","Data":"0b7ad5277b8472209ebfc568435438f7aa1fb5739fc8d1f25a128c2a020f59b1"}
Jul 01 17:19:16.706735 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:16.706716    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-52e43-predictor-674cd6cc9-prtkw" event={"ID":"430b61ee-095e-470f-a487-7d790277b37d","Type":"ContainerStarted","Data":"9c4643a1bdda6c42cd25206ebaee613f1b0f93ea2d9d1becea3c9dc259344dc7"}
Jul 01 17:19:16.706825 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:16.706757    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-52e43-predictor-674cd6cc9-prtkw" event={"ID":"430b61ee-095e-470f-a487-7d790277b37d","Type":"ContainerStarted","Data":"29703f53293de1a8ce91cec427afcda1e102f2a4eb75e24aa33249e583a500de"}
Jul 01 17:19:16.728068 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:16.728020    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-52e43-predictor-79768fc67d-xk27k" podStartSLOduration=1.728004093 podStartE2EDuration="1.728004093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:19:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:19:16.726399135 +0000 UTC m=+1274.523537564" watchObservedRunningTime="2026-07-01 17:19:16.728004093 +0000 UTC m=+1274.525142494"
Jul 01 17:19:17.437950 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:17.437907    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3c336-predictor-54665d447b-s2vc2" podUID="0ce82ad2-7236-475b-b3ff-eccfa49eccfa" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Jul 01 17:19:17.442330 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:17.442302    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3c336-predictor-54665d447b-s2vc2" podUID="0ce82ad2-7236-475b-b3ff-eccfa49eccfa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 01 17:19:17.715858 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:17.715774    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-52e43-predictor-674cd6cc9-prtkw" event={"ID":"430b61ee-095e-470f-a487-7d790277b37d","Type":"ContainerStarted","Data":"2cd75215f5ea925dd20a078b5135506896d83f8e825b469b6b0117b3d09edbb3"}
Jul 01 17:19:17.716272 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:17.716016    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-52e43-predictor-79768fc67d-xk27k"
Jul 01 17:19:17.717363 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:17.717315    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-52e43-predictor-674cd6cc9-prtkw"
Jul 01 17:19:17.717478 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:17.717393    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-52e43-predictor-674cd6cc9-prtkw"
Jul 01 17:19:17.717536 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:17.717475    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-52e43-predictor-674cd6cc9-prtkw" podUID="430b61ee-095e-470f-a487-7d790277b37d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 01 17:19:17.717599 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:17.717577    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-52e43-predictor-79768fc67d-xk27k" podUID="e49336bc-9319-467b-a6b8-969be03815e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 01 17:19:17.737486 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:17.737434    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-52e43-predictor-674cd6cc9-prtkw" podStartSLOduration=2.737419108 podStartE2EDuration="2.737419108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:19:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:19:17.736249992 +0000 UTC m=+1275.533388419" watchObservedRunningTime="2026-07-01 17:19:17.737419108 +0000 UTC m=+1275.534557508"
Jul 01 17:19:18.441093 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:18.441047    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3c336-predictor-547bb6bc8f-96z6b" podUID="dc760277-8744-4acb-89dc-4cbc32f83742" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.26:8643/healthz\": dial tcp 10.132.0.26:8643: connect: connection refused"
Jul 01 17:19:18.447611 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:18.447580    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3c336-predictor-547bb6bc8f-96z6b" podUID="dc760277-8744-4acb-89dc-4cbc32f83742" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 01 17:19:18.720429 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:18.720394    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-52e43-predictor-79768fc67d-xk27k" podUID="e49336bc-9319-467b-a6b8-969be03815e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 01 17:19:18.720805 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:18.720505    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-52e43-predictor-674cd6cc9-prtkw" podUID="430b61ee-095e-470f-a487-7d790277b37d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 01 17:19:19.124135 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:19.124109    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3c336-predictor-54665d447b-s2vc2"
Jul 01 17:19:19.222375 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:19.222310    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0ce82ad2-7236-475b-b3ff-eccfa49eccfa-kube-rbac-proxy-sar-config\") pod \"0ce82ad2-7236-475b-b3ff-eccfa49eccfa\" (UID: \"0ce82ad2-7236-475b-b3ff-eccfa49eccfa\") "
Jul 01 17:19:19.222588 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:19.222567    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m6xtq\" (UniqueName: \"kubernetes.io/projected/0ce82ad2-7236-475b-b3ff-eccfa49eccfa-kube-api-access-m6xtq\") pod \"0ce82ad2-7236-475b-b3ff-eccfa49eccfa\" (UID: \"0ce82ad2-7236-475b-b3ff-eccfa49eccfa\") "
Jul 01 17:19:19.222678 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:19.222620    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0ce82ad2-7236-475b-b3ff-eccfa49eccfa-proxy-tls\") pod \"0ce82ad2-7236-475b-b3ff-eccfa49eccfa\" (UID: \"0ce82ad2-7236-475b-b3ff-eccfa49eccfa\") "
Jul 01 17:19:19.222984 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:19.222958    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ce82ad2-7236-475b-b3ff-eccfa49eccfa-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0ce82ad2-7236-475b-b3ff-eccfa49eccfa" (UID: "0ce82ad2-7236-475b-b3ff-eccfa49eccfa"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:19:19.224803 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:19.224778    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ce82ad2-7236-475b-b3ff-eccfa49eccfa-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0ce82ad2-7236-475b-b3ff-eccfa49eccfa" (UID: "0ce82ad2-7236-475b-b3ff-eccfa49eccfa"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 17:19:19.224876 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:19.224845    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ce82ad2-7236-475b-b3ff-eccfa49eccfa-kube-api-access-m6xtq" (OuterVolumeSpecName: "kube-api-access-m6xtq") pod "0ce82ad2-7236-475b-b3ff-eccfa49eccfa" (UID: "0ce82ad2-7236-475b-b3ff-eccfa49eccfa"). InnerVolumeSpecName "kube-api-access-m6xtq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 17:19:19.324208 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:19.324177    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0ce82ad2-7236-475b-b3ff-eccfa49eccfa-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:19:19.324208 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:19.324207    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6xtq\" (UniqueName: \"kubernetes.io/projected/0ce82ad2-7236-475b-b3ff-eccfa49eccfa-kube-api-access-m6xtq\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:19:19.324447 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:19.324222    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0ce82ad2-7236-475b-b3ff-eccfa49eccfa-proxy-tls\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:19:19.366739 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:19.366716    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3c336-predictor-547bb6bc8f-96z6b"
Jul 01 17:19:19.425123 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:19.425087    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc760277-8744-4acb-89dc-4cbc32f83742-kube-rbac-proxy-sar-config\") pod \"dc760277-8744-4acb-89dc-4cbc32f83742\" (UID: \"dc760277-8744-4acb-89dc-4cbc32f83742\") "
Jul 01 17:19:19.425289 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:19.425136    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc760277-8744-4acb-89dc-4cbc32f83742-proxy-tls\") pod \"dc760277-8744-4acb-89dc-4cbc32f83742\" (UID: \"dc760277-8744-4acb-89dc-4cbc32f83742\") "
Jul 01 17:19:19.425359 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:19.425319    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sx7nz\" (UniqueName: \"kubernetes.io/projected/dc760277-8744-4acb-89dc-4cbc32f83742-kube-api-access-sx7nz\") pod \"dc760277-8744-4acb-89dc-4cbc32f83742\" (UID: \"dc760277-8744-4acb-89dc-4cbc32f83742\") "
Jul 01 17:19:19.425507 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:19.425481    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc760277-8744-4acb-89dc-4cbc32f83742-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "dc760277-8744-4acb-89dc-4cbc32f83742" (UID: "dc760277-8744-4acb-89dc-4cbc32f83742"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:19:19.425637 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:19.425618    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc760277-8744-4acb-89dc-4cbc32f83742-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:19:19.427369 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:19.427314    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc760277-8744-4acb-89dc-4cbc32f83742-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dc760277-8744-4acb-89dc-4cbc32f83742" (UID: "dc760277-8744-4acb-89dc-4cbc32f83742"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 17:19:19.427493 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:19.427476    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc760277-8744-4acb-89dc-4cbc32f83742-kube-api-access-sx7nz" (OuterVolumeSpecName: "kube-api-access-sx7nz") pod "dc760277-8744-4acb-89dc-4cbc32f83742" (UID: "dc760277-8744-4acb-89dc-4cbc32f83742"). InnerVolumeSpecName "kube-api-access-sx7nz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 17:19:19.527061 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:19.526977    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc760277-8744-4acb-89dc-4cbc32f83742-proxy-tls\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:19:19.527061 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:19.527006    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sx7nz\" (UniqueName: \"kubernetes.io/projected/dc760277-8744-4acb-89dc-4cbc32f83742-kube-api-access-sx7nz\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:19:19.726421 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:19.726387    2563 generic.go:356] "Generic (PLEG): container finished" podID="dc760277-8744-4acb-89dc-4cbc32f83742" containerID="9f335543f9b6ea288b41e81269025946946c16bb316bad568b859148f62e1686" exitCode=0
Jul 01 17:19:19.726851 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:19.726440    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3c336-predictor-547bb6bc8f-96z6b"
Jul 01 17:19:19.726851 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:19.726472    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3c336-predictor-547bb6bc8f-96z6b" event={"ID":"dc760277-8744-4acb-89dc-4cbc32f83742","Type":"ContainerDied","Data":"9f335543f9b6ea288b41e81269025946946c16bb316bad568b859148f62e1686"}
Jul 01 17:19:19.726851 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:19.726527    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3c336-predictor-547bb6bc8f-96z6b" event={"ID":"dc760277-8744-4acb-89dc-4cbc32f83742","Type":"ContainerDied","Data":"a9341343c538a35b8a36009181027ece843b15801c84737b604b28cd9ac725e1"}
Jul 01 17:19:19.726851 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:19.726560    2563 scope.go:117] "RemoveContainer" containerID="c0d752993c86d6c7a8d349f2444a6d7030def616cbc34f7cb85a847cd7ada222"
Jul 01 17:19:19.728327 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:19.728305    2563 generic.go:356] "Generic (PLEG): container finished" podID="0ce82ad2-7236-475b-b3ff-eccfa49eccfa" containerID="624b66f1b669b1173df86310452a5c315ded3f96a559c446c912fd16e2a1723b" exitCode=0
Jul 01 17:19:19.728506 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:19.728381    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3c336-predictor-54665d447b-s2vc2" event={"ID":"0ce82ad2-7236-475b-b3ff-eccfa49eccfa","Type":"ContainerDied","Data":"624b66f1b669b1173df86310452a5c315ded3f96a559c446c912fd16e2a1723b"}
Jul 01 17:19:19.728506 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:19.728394    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3c336-predictor-54665d447b-s2vc2"
Jul 01 17:19:19.728506 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:19.728432    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3c336-predictor-54665d447b-s2vc2" event={"ID":"0ce82ad2-7236-475b-b3ff-eccfa49eccfa","Type":"ContainerDied","Data":"edc723769dc8f39359aecc8c3f2ec68e2820f24daee3a2bcab24bd2fda8e984a"}
Jul 01 17:19:19.729063 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:19.729035    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-52e43-predictor-674cd6cc9-prtkw" podUID="430b61ee-095e-470f-a487-7d790277b37d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 01 17:19:19.734965 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:19.734949    2563 scope.go:117] "RemoveContainer" containerID="9f335543f9b6ea288b41e81269025946946c16bb316bad568b859148f62e1686"
Jul 01 17:19:19.743147 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:19.743129    2563 scope.go:117] "RemoveContainer" containerID="c0d752993c86d6c7a8d349f2444a6d7030def616cbc34f7cb85a847cd7ada222"
Jul 01 17:19:19.743461 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:19:19.743437    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0d752993c86d6c7a8d349f2444a6d7030def616cbc34f7cb85a847cd7ada222\": container with ID starting with c0d752993c86d6c7a8d349f2444a6d7030def616cbc34f7cb85a847cd7ada222 not found: ID does not exist" containerID="c0d752993c86d6c7a8d349f2444a6d7030def616cbc34f7cb85a847cd7ada222"
Jul 01 17:19:19.743577 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:19.743470    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0d752993c86d6c7a8d349f2444a6d7030def616cbc34f7cb85a847cd7ada222"} err="failed to get container status \"c0d752993c86d6c7a8d349f2444a6d7030def616cbc34f7cb85a847cd7ada222\": rpc error: code = NotFound desc = could not find container \"c0d752993c86d6c7a8d349f2444a6d7030def616cbc34f7cb85a847cd7ada222\": container with ID starting with c0d752993c86d6c7a8d349f2444a6d7030def616cbc34f7cb85a847cd7ada222 not found: ID does not exist"
Jul 01 17:19:19.743577 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:19.743487    2563 scope.go:117] "RemoveContainer" containerID="9f335543f9b6ea288b41e81269025946946c16bb316bad568b859148f62e1686"
Jul 01 17:19:19.743782 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:19:19.743761    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f335543f9b6ea288b41e81269025946946c16bb316bad568b859148f62e1686\": container with ID starting with 9f335543f9b6ea288b41e81269025946946c16bb316bad568b859148f62e1686 not found: ID does not exist" containerID="9f335543f9b6ea288b41e81269025946946c16bb316bad568b859148f62e1686"
Jul 01 17:19:19.743818 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:19.743788    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f335543f9b6ea288b41e81269025946946c16bb316bad568b859148f62e1686"} err="failed to get container status \"9f335543f9b6ea288b41e81269025946946c16bb316bad568b859148f62e1686\": rpc error: code = NotFound desc = could not find container \"9f335543f9b6ea288b41e81269025946946c16bb316bad568b859148f62e1686\": container with ID starting with 9f335543f9b6ea288b41e81269025946946c16bb316bad568b859148f62e1686 not found: ID does not exist"
Jul 01 17:19:19.743818 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:19.743804    2563 scope.go:117] "RemoveContainer" containerID="0b7ad5277b8472209ebfc568435438f7aa1fb5739fc8d1f25a128c2a020f59b1"
Jul 01 17:19:19.751306 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:19.751289    2563 scope.go:117] "RemoveContainer" containerID="624b66f1b669b1173df86310452a5c315ded3f96a559c446c912fd16e2a1723b"
Jul 01 17:19:19.754490 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:19.754467    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3c336-predictor-547bb6bc8f-96z6b"]
Jul 01 17:19:19.758954 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:19.758934    2563 scope.go:117] "RemoveContainer" containerID="0b7ad5277b8472209ebfc568435438f7aa1fb5739fc8d1f25a128c2a020f59b1"
Jul 01 17:19:19.759426 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:19:19.759385    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b7ad5277b8472209ebfc568435438f7aa1fb5739fc8d1f25a128c2a020f59b1\": container with ID starting with 0b7ad5277b8472209ebfc568435438f7aa1fb5739fc8d1f25a128c2a020f59b1 not found: ID does not exist" containerID="0b7ad5277b8472209ebfc568435438f7aa1fb5739fc8d1f25a128c2a020f59b1"
Jul 01 17:19:19.759683 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:19.759419    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b7ad5277b8472209ebfc568435438f7aa1fb5739fc8d1f25a128c2a020f59b1"} err="failed to get container status \"0b7ad5277b8472209ebfc568435438f7aa1fb5739fc8d1f25a128c2a020f59b1\": rpc error: code = NotFound desc = could not find container \"0b7ad5277b8472209ebfc568435438f7aa1fb5739fc8d1f25a128c2a020f59b1\": container with ID starting with 0b7ad5277b8472209ebfc568435438f7aa1fb5739fc8d1f25a128c2a020f59b1 not found: ID does not exist"
Jul 01 17:19:19.759683 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:19.759441    2563 scope.go:117] "RemoveContainer" containerID="624b66f1b669b1173df86310452a5c315ded3f96a559c446c912fd16e2a1723b"
Jul 01 17:19:19.760278 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:19:19.760255    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"624b66f1b669b1173df86310452a5c315ded3f96a559c446c912fd16e2a1723b\": container with ID starting with 624b66f1b669b1173df86310452a5c315ded3f96a559c446c912fd16e2a1723b not found: ID does not exist" containerID="624b66f1b669b1173df86310452a5c315ded3f96a559c446c912fd16e2a1723b"
Jul 01 17:19:19.760365 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:19.760287    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"624b66f1b669b1173df86310452a5c315ded3f96a559c446c912fd16e2a1723b"} err="failed to get container status \"624b66f1b669b1173df86310452a5c315ded3f96a559c446c912fd16e2a1723b\": rpc error: code = NotFound desc = could not find container \"624b66f1b669b1173df86310452a5c315ded3f96a559c446c912fd16e2a1723b\": container with ID starting with 624b66f1b669b1173df86310452a5c315ded3f96a559c446c912fd16e2a1723b not found: ID does not exist"
Jul 01 17:19:19.761653 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:19.761638    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3c336-predictor-547bb6bc8f-96z6b"]
Jul 01 17:19:19.773593 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:19.773570    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3c336-predictor-54665d447b-s2vc2"]
Jul 01 17:19:19.776311 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:19.776292    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3c336-predictor-54665d447b-s2vc2"]
Jul 01 17:19:20.696860 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:20.696832    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ce82ad2-7236-475b-b3ff-eccfa49eccfa" path="/var/lib/kubelet/pods/0ce82ad2-7236-475b-b3ff-eccfa49eccfa/volumes"
Jul 01 17:19:20.697232 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:20.697220    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc760277-8744-4acb-89dc-4cbc32f83742" path="/var/lib/kubelet/pods/dc760277-8744-4acb-89dc-4cbc32f83742/volumes"
Jul 01 17:19:23.725720 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:23.725694    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-52e43-predictor-79768fc67d-xk27k"
Jul 01 17:19:23.726209 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:23.726184    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-52e43-predictor-79768fc67d-xk27k" podUID="e49336bc-9319-467b-a6b8-969be03815e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 01 17:19:24.733910 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:24.733885    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-52e43-predictor-674cd6cc9-prtkw"
Jul 01 17:19:24.734407 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:24.734381    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-52e43-predictor-674cd6cc9-prtkw" podUID="430b61ee-095e-470f-a487-7d790277b37d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 01 17:19:31.255962 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:31.255913    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:19:31.354478 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:31.354446    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ce016-predictor-6ffdb68b79-hq49w"]
Jul 01 17:19:31.354735 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:31.354707    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-ce016-predictor-6ffdb68b79-hq49w" podUID="bee5b713-8b32-437d-86ad-c6c646b805c3" containerName="kserve-container" containerID="cri-o://8973fa6bb27fafb35d84a407e0c752d080de1f6fb7fe9fb4c5e7804030c8a1d9" gracePeriod=30
Jul 01 17:19:31.354862 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:31.354754    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-ce016-predictor-6ffdb68b79-hq49w" podUID="bee5b713-8b32-437d-86ad-c6c646b805c3" containerName="kube-rbac-proxy" containerID="cri-o://c391b181f049b8884f6b34bdd1c754b113af19c280b9a33df7d1552395692064" gracePeriod=30
Jul 01 17:19:31.382510 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:31.382474    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:19:31.384964 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:31.384938    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1f194-predictor-6dd685749f-bq5w2"]
Jul 01 17:19:31.385559 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:31.385532    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc760277-8744-4acb-89dc-4cbc32f83742" containerName="kube-rbac-proxy"
Jul 01 17:19:31.385660 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:31.385561    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc760277-8744-4acb-89dc-4cbc32f83742" containerName="kube-rbac-proxy"
Jul 01 17:19:31.385660 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:31.385583    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc760277-8744-4acb-89dc-4cbc32f83742" containerName="kserve-container"
Jul 01 17:19:31.385660 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:31.385591    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc760277-8744-4acb-89dc-4cbc32f83742" containerName="kserve-container"
Jul 01 17:19:31.385660 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:31.385609    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ce82ad2-7236-475b-b3ff-eccfa49eccfa" containerName="kserve-container"
Jul 01 17:19:31.385660 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:31.385618    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ce82ad2-7236-475b-b3ff-eccfa49eccfa" containerName="kserve-container"
Jul 01 17:19:31.385660 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:31.385643    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ce82ad2-7236-475b-b3ff-eccfa49eccfa" containerName="kube-rbac-proxy"
Jul 01 17:19:31.385660 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:31.385651    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ce82ad2-7236-475b-b3ff-eccfa49eccfa" containerName="kube-rbac-proxy"
Jul 01 17:19:31.385903 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:31.385719    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ce82ad2-7236-475b-b3ff-eccfa49eccfa" containerName="kserve-container"
Jul 01 17:19:31.385903 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:31.385729    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ce82ad2-7236-475b-b3ff-eccfa49eccfa" containerName="kube-rbac-proxy"
Jul 01 17:19:31.385903 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:31.385737    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc760277-8744-4acb-89dc-4cbc32f83742" containerName="kube-rbac-proxy"
Jul 01 17:19:31.385903 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:31.385743    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc760277-8744-4acb-89dc-4cbc32f83742" containerName="kserve-container"
Jul 01 17:19:31.388906 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:31.388891    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-1f194-predictor-6dd685749f-bq5w2"
Jul 01 17:19:31.391794 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:31.391770    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-1f194-predictor-serving-cert\""
Jul 01 17:19:31.391794 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:31.391786    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-1f194-kube-rbac-proxy-sar-config\""
Jul 01 17:19:31.400958 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:31.400937    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1f194-predictor-6dd685749f-bq5w2"]
Jul 01 17:19:31.426548 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:31.426518    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/acb61295-6868-4bc3-9a75-77fae13527fd-proxy-tls\") pod \"success-200-isvc-1f194-predictor-6dd685749f-bq5w2\" (UID: \"acb61295-6868-4bc3-9a75-77fae13527fd\") " pod="kserve-ci-e2e-test/success-200-isvc-1f194-predictor-6dd685749f-bq5w2"
Jul 01 17:19:31.426723 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:31.426556    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/acb61295-6868-4bc3-9a75-77fae13527fd-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-1f194-predictor-6dd685749f-bq5w2\" (UID: \"acb61295-6868-4bc3-9a75-77fae13527fd\") " pod="kserve-ci-e2e-test/success-200-isvc-1f194-predictor-6dd685749f-bq5w2"
Jul 01 17:19:31.426723 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:31.426623    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8w2w\" (UniqueName: \"kubernetes.io/projected/acb61295-6868-4bc3-9a75-77fae13527fd-kube-api-access-r8w2w\") pod \"success-200-isvc-1f194-predictor-6dd685749f-bq5w2\" (UID: \"acb61295-6868-4bc3-9a75-77fae13527fd\") " pod="kserve-ci-e2e-test/success-200-isvc-1f194-predictor-6dd685749f-bq5w2"
Jul 01 17:19:31.479840 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:31.479811    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:19:31.493839 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:31.493814    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1f194-predictor-6c5d4fc46b-kk5sz"]
Jul 01 17:19:31.497226 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:31.497211    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1f194-predictor-6c5d4fc46b-kk5sz"
Jul 01 17:19:31.500196 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:31.500169    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-1f194-predictor-serving-cert\""
Jul 01 17:19:31.500196 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:31.500194    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-1f194-kube-rbac-proxy-sar-config\""
Jul 01 17:19:31.509727 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:31.509679    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1f194-predictor-6c5d4fc46b-kk5sz"]
Jul 01 17:19:31.527766 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:31.527737    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/acb61295-6868-4bc3-9a75-77fae13527fd-proxy-tls\") pod \"success-200-isvc-1f194-predictor-6dd685749f-bq5w2\" (UID: \"acb61295-6868-4bc3-9a75-77fae13527fd\") " pod="kserve-ci-e2e-test/success-200-isvc-1f194-predictor-6dd685749f-bq5w2"
Jul 01 17:19:31.527909 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:31.527776    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9jhc\" (UniqueName: \"kubernetes.io/projected/dc2febf1-bc84-4180-8f3c-da46de283e1b-kube-api-access-k9jhc\") pod \"error-404-isvc-1f194-predictor-6c5d4fc46b-kk5sz\" (UID: \"dc2febf1-bc84-4180-8f3c-da46de283e1b\") " pod="kserve-ci-e2e-test/error-404-isvc-1f194-predictor-6c5d4fc46b-kk5sz"
Jul 01 17:19:31.527909 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:31.527800    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/acb61295-6868-4bc3-9a75-77fae13527fd-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-1f194-predictor-6dd685749f-bq5w2\" (UID: \"acb61295-6868-4bc3-9a75-77fae13527fd\") " pod="kserve-ci-e2e-test/success-200-isvc-1f194-predictor-6dd685749f-bq5w2"
Jul 01 17:19:31.527909 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:31.527824    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc2febf1-bc84-4180-8f3c-da46de283e1b-proxy-tls\") pod \"error-404-isvc-1f194-predictor-6c5d4fc46b-kk5sz\" (UID: \"dc2febf1-bc84-4180-8f3c-da46de283e1b\") " pod="kserve-ci-e2e-test/error-404-isvc-1f194-predictor-6c5d4fc46b-kk5sz"
Jul 01 17:19:31.527909 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:31.527887    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8w2w\" (UniqueName: \"kubernetes.io/projected/acb61295-6868-4bc3-9a75-77fae13527fd-kube-api-access-r8w2w\") pod \"success-200-isvc-1f194-predictor-6dd685749f-bq5w2\" (UID: \"acb61295-6868-4bc3-9a75-77fae13527fd\") " pod="kserve-ci-e2e-test/success-200-isvc-1f194-predictor-6dd685749f-bq5w2"
Jul 01 17:19:31.527909 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:19:31.527901    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-1f194-predictor-serving-cert: secret "success-200-isvc-1f194-predictor-serving-cert" not found
Jul 01 17:19:31.528320 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:19:31.527968    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acb61295-6868-4bc3-9a75-77fae13527fd-proxy-tls podName:acb61295-6868-4bc3-9a75-77fae13527fd nodeName:}" failed. No retries permitted until 2026-07-01 17:19:32.027947922 +0000 UTC m=+1289.825086321 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/acb61295-6868-4bc3-9a75-77fae13527fd-proxy-tls") pod "success-200-isvc-1f194-predictor-6dd685749f-bq5w2" (UID: "acb61295-6868-4bc3-9a75-77fae13527fd") : secret "success-200-isvc-1f194-predictor-serving-cert" not found
Jul 01 17:19:31.528320 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:31.527994    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc2febf1-bc84-4180-8f3c-da46de283e1b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-1f194-predictor-6c5d4fc46b-kk5sz\" (UID: \"dc2febf1-bc84-4180-8f3c-da46de283e1b\") " pod="kserve-ci-e2e-test/error-404-isvc-1f194-predictor-6c5d4fc46b-kk5sz"
Jul 01 17:19:31.528414 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:31.528392    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/acb61295-6868-4bc3-9a75-77fae13527fd-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-1f194-predictor-6dd685749f-bq5w2\" (UID: \"acb61295-6868-4bc3-9a75-77fae13527fd\") " pod="kserve-ci-e2e-test/success-200-isvc-1f194-predictor-6dd685749f-bq5w2"
Jul 01 17:19:31.540459 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:31.540436    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8w2w\" (UniqueName: \"kubernetes.io/projected/acb61295-6868-4bc3-9a75-77fae13527fd-kube-api-access-r8w2w\") pod \"success-200-isvc-1f194-predictor-6dd685749f-bq5w2\" (UID: \"acb61295-6868-4bc3-9a75-77fae13527fd\") " pod="kserve-ci-e2e-test/success-200-isvc-1f194-predictor-6dd685749f-bq5w2"
Jul 01 17:19:31.567724 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:31.567695    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ce016-predictor-84dbff7cdb-rdb8n"]
Jul 01 17:19:31.567954 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:31.567921    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-ce016-predictor-84dbff7cdb-rdb8n" podUID="318c4e47-af61-4745-be04-a6ca2637ed1e" containerName="kserve-container" containerID="cri-o://57712d8baa361854be1ce1ba6abc303567ea7897a4316e2078a49c767c3b38ef" gracePeriod=30
Jul 01 17:19:31.568121 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:31.567963    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-ce016-predictor-84dbff7cdb-rdb8n" podUID="318c4e47-af61-4745-be04-a6ca2637ed1e" containerName="kube-rbac-proxy" containerID="cri-o://ccdf69b4fd13e204738793a3890ee4b0ed6e8d50d5334d772710f529dab71379" gracePeriod=30
Jul 01 17:19:31.574012 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:31.573571    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:19:31.629438 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:31.629388    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc2febf1-bc84-4180-8f3c-da46de283e1b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-1f194-predictor-6c5d4fc46b-kk5sz\" (UID: \"dc2febf1-bc84-4180-8f3c-da46de283e1b\") " pod="kserve-ci-e2e-test/error-404-isvc-1f194-predictor-6c5d4fc46b-kk5sz"
Jul 01 17:19:31.629602 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:31.629490    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9jhc\" (UniqueName: \"kubernetes.io/projected/dc2febf1-bc84-4180-8f3c-da46de283e1b-kube-api-access-k9jhc\") pod \"error-404-isvc-1f194-predictor-6c5d4fc46b-kk5sz\" (UID: \"dc2febf1-bc84-4180-8f3c-da46de283e1b\") " pod="kserve-ci-e2e-test/error-404-isvc-1f194-predictor-6c5d4fc46b-kk5sz"
Jul 01 17:19:31.629602 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:31.629519    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc2febf1-bc84-4180-8f3c-da46de283e1b-proxy-tls\") pod \"error-404-isvc-1f194-predictor-6c5d4fc46b-kk5sz\" (UID: \"dc2febf1-bc84-4180-8f3c-da46de283e1b\") " pod="kserve-ci-e2e-test/error-404-isvc-1f194-predictor-6c5d4fc46b-kk5sz"
Jul 01 17:19:31.629727 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:19:31.629679    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-1f194-predictor-serving-cert: secret "error-404-isvc-1f194-predictor-serving-cert" not found
Jul 01 17:19:31.629779 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:19:31.629755    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc2febf1-bc84-4180-8f3c-da46de283e1b-proxy-tls podName:dc2febf1-bc84-4180-8f3c-da46de283e1b nodeName:}" failed. No retries permitted until 2026-07-01 17:19:32.129736682 +0000 UTC m=+1289.926875060 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/dc2febf1-bc84-4180-8f3c-da46de283e1b-proxy-tls") pod "error-404-isvc-1f194-predictor-6c5d4fc46b-kk5sz" (UID: "dc2febf1-bc84-4180-8f3c-da46de283e1b") : secret "error-404-isvc-1f194-predictor-serving-cert" not found
Jul 01 17:19:31.630047 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:31.630027    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc2febf1-bc84-4180-8f3c-da46de283e1b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-1f194-predictor-6c5d4fc46b-kk5sz\" (UID: \"dc2febf1-bc84-4180-8f3c-da46de283e1b\") " pod="kserve-ci-e2e-test/error-404-isvc-1f194-predictor-6c5d4fc46b-kk5sz"
Jul 01 17:19:31.639835 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:31.639809    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9jhc\" (UniqueName: \"kubernetes.io/projected/dc2febf1-bc84-4180-8f3c-da46de283e1b-kube-api-access-k9jhc\") pod \"error-404-isvc-1f194-predictor-6c5d4fc46b-kk5sz\" (UID: \"dc2febf1-bc84-4180-8f3c-da46de283e1b\") " pod="kserve-ci-e2e-test/error-404-isvc-1f194-predictor-6c5d4fc46b-kk5sz"
Jul 01 17:19:31.653395 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:31.653372    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:19:31.764181 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:31.764097    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:19:31.782811 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:31.782779    2563 generic.go:356] "Generic (PLEG): container finished" podID="318c4e47-af61-4745-be04-a6ca2637ed1e" containerID="ccdf69b4fd13e204738793a3890ee4b0ed6e8d50d5334d772710f529dab71379" exitCode=2
Jul 01 17:19:31.782972 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:31.782857    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ce016-predictor-84dbff7cdb-rdb8n" event={"ID":"318c4e47-af61-4745-be04-a6ca2637ed1e","Type":"ContainerDied","Data":"ccdf69b4fd13e204738793a3890ee4b0ed6e8d50d5334d772710f529dab71379"}
Jul 01 17:19:31.784671 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:31.784645    2563 generic.go:356] "Generic (PLEG): container finished" podID="bee5b713-8b32-437d-86ad-c6c646b805c3" containerID="c391b181f049b8884f6b34bdd1c754b113af19c280b9a33df7d1552395692064" exitCode=2
Jul 01 17:19:31.784789 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:31.784712    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ce016-predictor-6ffdb68b79-hq49w" event={"ID":"bee5b713-8b32-437d-86ad-c6c646b805c3","Type":"ContainerDied","Data":"c391b181f049b8884f6b34bdd1c754b113af19c280b9a33df7d1552395692064"}
Jul 01 17:19:32.033720 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:32.033621    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/acb61295-6868-4bc3-9a75-77fae13527fd-proxy-tls\") pod \"success-200-isvc-1f194-predictor-6dd685749f-bq5w2\" (UID: \"acb61295-6868-4bc3-9a75-77fae13527fd\") " pod="kserve-ci-e2e-test/success-200-isvc-1f194-predictor-6dd685749f-bq5w2"
Jul 01 17:19:32.036193 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:32.036174    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/acb61295-6868-4bc3-9a75-77fae13527fd-proxy-tls\") pod \"success-200-isvc-1f194-predictor-6dd685749f-bq5w2\" (UID: \"acb61295-6868-4bc3-9a75-77fae13527fd\") " pod="kserve-ci-e2e-test/success-200-isvc-1f194-predictor-6dd685749f-bq5w2"
Jul 01 17:19:32.059672 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:32.059640    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:19:32.134298 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:32.134257    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc2febf1-bc84-4180-8f3c-da46de283e1b-proxy-tls\") pod \"error-404-isvc-1f194-predictor-6c5d4fc46b-kk5sz\" (UID: \"dc2febf1-bc84-4180-8f3c-da46de283e1b\") " pod="kserve-ci-e2e-test/error-404-isvc-1f194-predictor-6c5d4fc46b-kk5sz"
Jul 01 17:19:32.136776 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:32.136756    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc2febf1-bc84-4180-8f3c-da46de283e1b-proxy-tls\") pod \"error-404-isvc-1f194-predictor-6c5d4fc46b-kk5sz\" (UID: \"dc2febf1-bc84-4180-8f3c-da46de283e1b\") " pod="kserve-ci-e2e-test/error-404-isvc-1f194-predictor-6c5d4fc46b-kk5sz"
Jul 01 17:19:32.299903 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:32.299814    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-1f194-predictor-6dd685749f-bq5w2"
Jul 01 17:19:32.408009 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:32.407974    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1f194-predictor-6c5d4fc46b-kk5sz"
Jul 01 17:19:32.440859 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:32.440829    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1f194-predictor-6dd685749f-bq5w2"]
Jul 01 17:19:32.443214 ip-10-0-137-180 kubenswrapper[2563]: W0701 17:19:32.443183    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacb61295_6868_4bc3_9a75_77fae13527fd.slice/crio-95816f29708bab823c35b1bd27d46553c014bae521f6bb70a034e34d3518c2f2 WatchSource:0}: Error finding container 95816f29708bab823c35b1bd27d46553c014bae521f6bb70a034e34d3518c2f2: Status 404 returned error can't find the container with id 95816f29708bab823c35b1bd27d46553c014bae521f6bb70a034e34d3518c2f2
Jul 01 17:19:32.559849 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:32.559825    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1f194-predictor-6c5d4fc46b-kk5sz"]
Jul 01 17:19:32.560432 ip-10-0-137-180 kubenswrapper[2563]: W0701 17:19:32.560403    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc2febf1_bc84_4180_8f3c_da46de283e1b.slice/crio-394104227d0f9e5eaa5f79afeb1d4a30657fcf8b255af02454ce980608ca189f WatchSource:0}: Error finding container 394104227d0f9e5eaa5f79afeb1d4a30657fcf8b255af02454ce980608ca189f: Status 404 returned error can't find the container with id 394104227d0f9e5eaa5f79afeb1d4a30657fcf8b255af02454ce980608ca189f
Jul 01 17:19:32.792133 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:32.792096    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1f194-predictor-6c5d4fc46b-kk5sz" event={"ID":"dc2febf1-bc84-4180-8f3c-da46de283e1b","Type":"ContainerStarted","Data":"34c308d4f10809a65bf679816d0f1b7e41d54324e6c09ed016a9236ee3d58cce"}
Jul 01 17:19:32.792308 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:32.792160    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1f194-predictor-6c5d4fc46b-kk5sz" event={"ID":"dc2febf1-bc84-4180-8f3c-da46de283e1b","Type":"ContainerStarted","Data":"921fa68fcb3fcb73af90ff3ad0ee836e579e16f40df0df682dc03fc16d995ccd"}
Jul 01 17:19:32.792308 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:32.792187    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-1f194-predictor-6c5d4fc46b-kk5sz"
Jul 01 17:19:32.792308 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:32.792219    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1f194-predictor-6c5d4fc46b-kk5sz" event={"ID":"dc2febf1-bc84-4180-8f3c-da46de283e1b","Type":"ContainerStarted","Data":"394104227d0f9e5eaa5f79afeb1d4a30657fcf8b255af02454ce980608ca189f"}
Jul 01 17:19:32.792308 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:32.792243    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-1f194-predictor-6c5d4fc46b-kk5sz"
Jul 01 17:19:32.793838 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:32.793807    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1f194-predictor-6c5d4fc46b-kk5sz" podUID="dc2febf1-bc84-4180-8f3c-da46de283e1b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 01 17:19:32.794120 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:32.794101    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1f194-predictor-6dd685749f-bq5w2" event={"ID":"acb61295-6868-4bc3-9a75-77fae13527fd","Type":"ContainerStarted","Data":"72cd997375fd6d06c89d1e69bfdcc076823292d2817df9bb3ccbd10175b1fa91"}
Jul 01 17:19:32.794178 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:32.794148    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-1f194-predictor-6dd685749f-bq5w2"
Jul 01 17:19:32.794178 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:32.794168    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1f194-predictor-6dd685749f-bq5w2" event={"ID":"acb61295-6868-4bc3-9a75-77fae13527fd","Type":"ContainerStarted","Data":"9e9afb6d6ed4a5b1219988d766ce10784798421fda448f6f7ac948eb8fed23ed"}
Jul 01 17:19:32.794259 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:32.794187    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1f194-predictor-6dd685749f-bq5w2" event={"ID":"acb61295-6868-4bc3-9a75-77fae13527fd","Type":"ContainerStarted","Data":"95816f29708bab823c35b1bd27d46553c014bae521f6bb70a034e34d3518c2f2"}
Jul 01 17:19:32.794259 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:32.794206    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-1f194-predictor-6dd685749f-bq5w2"
Jul 01 17:19:32.795290 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:32.795267    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1f194-predictor-6dd685749f-bq5w2" podUID="acb61295-6868-4bc3-9a75-77fae13527fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 01 17:19:32.834041 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:32.833920    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-1f194-predictor-6c5d4fc46b-kk5sz" podStartSLOduration=1.8338955380000002 podStartE2EDuration="1.833895538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:19:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:19:32.814765664 +0000 UTC m=+1290.611904079" watchObservedRunningTime="2026-07-01 17:19:32.833895538 +0000 UTC m=+1290.631033940"
Jul 01 17:19:32.834978 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:32.834940    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-1f194-predictor-6dd685749f-bq5w2" podStartSLOduration=1.834926289 podStartE2EDuration="1.834926289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:19:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:19:32.833313387 +0000 UTC m=+1290.630451813" watchObservedRunningTime="2026-07-01 17:19:32.834926289 +0000 UTC m=+1290.632064690"
Jul 01 17:19:33.726906 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:33.726870    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-52e43-predictor-79768fc67d-xk27k" podUID="e49336bc-9319-467b-a6b8-969be03815e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 01 17:19:33.798197 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:33.798158    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1f194-predictor-6c5d4fc46b-kk5sz" podUID="dc2febf1-bc84-4180-8f3c-da46de283e1b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 01 17:19:33.798370 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:33.798231    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1f194-predictor-6dd685749f-bq5w2" podUID="acb61295-6868-4bc3-9a75-77fae13527fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 01 17:19:34.562597 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:34.562555    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ce016-predictor-84dbff7cdb-rdb8n" podUID="318c4e47-af61-4745-be04-a6ca2637ed1e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Jul 01 17:19:34.568315 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:34.568293    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ce016-predictor-84dbff7cdb-rdb8n" podUID="318c4e47-af61-4745-be04-a6ca2637ed1e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 01 17:19:34.734706 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:34.734661    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-52e43-predictor-674cd6cc9-prtkw" podUID="430b61ee-095e-470f-a487-7d790277b37d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 01 17:19:35.519843 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:35.519809    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ce016-predictor-6ffdb68b79-hq49w"
Jul 01 17:19:35.565849 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:35.565767    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bee5b713-8b32-437d-86ad-c6c646b805c3-kube-rbac-proxy-sar-config\") pod \"bee5b713-8b32-437d-86ad-c6c646b805c3\" (UID: \"bee5b713-8b32-437d-86ad-c6c646b805c3\") "
Jul 01 17:19:35.565849 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:35.565808    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bee5b713-8b32-437d-86ad-c6c646b805c3-proxy-tls\") pod \"bee5b713-8b32-437d-86ad-c6c646b805c3\" (UID: \"bee5b713-8b32-437d-86ad-c6c646b805c3\") "
Jul 01 17:19:35.566068 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:35.565900    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r9t2c\" (UniqueName: \"kubernetes.io/projected/bee5b713-8b32-437d-86ad-c6c646b805c3-kube-api-access-r9t2c\") pod \"bee5b713-8b32-437d-86ad-c6c646b805c3\" (UID: \"bee5b713-8b32-437d-86ad-c6c646b805c3\") "
Jul 01 17:19:35.566159 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:35.566127    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bee5b713-8b32-437d-86ad-c6c646b805c3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bee5b713-8b32-437d-86ad-c6c646b805c3" (UID: "bee5b713-8b32-437d-86ad-c6c646b805c3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:19:35.568521 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:35.568488    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bee5b713-8b32-437d-86ad-c6c646b805c3-kube-api-access-r9t2c" (OuterVolumeSpecName: "kube-api-access-r9t2c") pod "bee5b713-8b32-437d-86ad-c6c646b805c3" (UID: "bee5b713-8b32-437d-86ad-c6c646b805c3"). InnerVolumeSpecName "kube-api-access-r9t2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 17:19:35.568521 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:35.568484    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bee5b713-8b32-437d-86ad-c6c646b805c3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bee5b713-8b32-437d-86ad-c6c646b805c3" (UID: "bee5b713-8b32-437d-86ad-c6c646b805c3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 17:19:35.595730 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:35.595708    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-ce016-predictor-84dbff7cdb-rdb8n"
Jul 01 17:19:35.666871 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:35.666845    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/318c4e47-af61-4745-be04-a6ca2637ed1e-kube-rbac-proxy-sar-config\") pod \"318c4e47-af61-4745-be04-a6ca2637ed1e\" (UID: \"318c4e47-af61-4745-be04-a6ca2637ed1e\") "
Jul 01 17:19:35.667062 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:35.666888    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/318c4e47-af61-4745-be04-a6ca2637ed1e-proxy-tls\") pod \"318c4e47-af61-4745-be04-a6ca2637ed1e\" (UID: \"318c4e47-af61-4745-be04-a6ca2637ed1e\") "
Jul 01 17:19:35.667062 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:35.666917    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bkxnz\" (UniqueName: \"kubernetes.io/projected/318c4e47-af61-4745-be04-a6ca2637ed1e-kube-api-access-bkxnz\") pod \"318c4e47-af61-4745-be04-a6ca2637ed1e\" (UID: \"318c4e47-af61-4745-be04-a6ca2637ed1e\") "
Jul 01 17:19:35.667202 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:35.667183    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9t2c\" (UniqueName: \"kubernetes.io/projected/bee5b713-8b32-437d-86ad-c6c646b805c3-kube-api-access-r9t2c\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:19:35.667265 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:35.667208    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bee5b713-8b32-437d-86ad-c6c646b805c3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:19:35.667265 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:35.667224    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bee5b713-8b32-437d-86ad-c6c646b805c3-proxy-tls\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:19:35.667265 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:35.667247    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/318c4e47-af61-4745-be04-a6ca2637ed1e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "318c4e47-af61-4745-be04-a6ca2637ed1e" (UID: "318c4e47-af61-4745-be04-a6ca2637ed1e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:19:35.669106 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:35.669087    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/318c4e47-af61-4745-be04-a6ca2637ed1e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "318c4e47-af61-4745-be04-a6ca2637ed1e" (UID: "318c4e47-af61-4745-be04-a6ca2637ed1e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 17:19:35.669153 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:35.669130    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/318c4e47-af61-4745-be04-a6ca2637ed1e-kube-api-access-bkxnz" (OuterVolumeSpecName: "kube-api-access-bkxnz") pod "318c4e47-af61-4745-be04-a6ca2637ed1e" (UID: "318c4e47-af61-4745-be04-a6ca2637ed1e"). InnerVolumeSpecName "kube-api-access-bkxnz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 17:19:35.768454 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:35.768418    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/318c4e47-af61-4745-be04-a6ca2637ed1e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:19:35.768454 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:35.768447    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/318c4e47-af61-4745-be04-a6ca2637ed1e-proxy-tls\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:19:35.768454 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:35.768458    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkxnz\" (UniqueName: \"kubernetes.io/projected/318c4e47-af61-4745-be04-a6ca2637ed1e-kube-api-access-bkxnz\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:19:35.808793 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:35.808758    2563 generic.go:356] "Generic (PLEG): container finished" podID="bee5b713-8b32-437d-86ad-c6c646b805c3" containerID="8973fa6bb27fafb35d84a407e0c752d080de1f6fb7fe9fb4c5e7804030c8a1d9" exitCode=0
Jul 01 17:19:35.808936 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:35.808876    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ce016-predictor-6ffdb68b79-hq49w" event={"ID":"bee5b713-8b32-437d-86ad-c6c646b805c3","Type":"ContainerDied","Data":"8973fa6bb27fafb35d84a407e0c752d080de1f6fb7fe9fb4c5e7804030c8a1d9"}
Jul 01 17:19:35.808936 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:35.808921    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ce016-predictor-6ffdb68b79-hq49w" event={"ID":"bee5b713-8b32-437d-86ad-c6c646b805c3","Type":"ContainerDied","Data":"a01535f9af0e3b38a865602643438b91b609ffef0c514f96db7dbe39ecc227d8"}
Jul 01 17:19:35.809028 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:35.808946    2563 scope.go:117] "RemoveContainer" containerID="c391b181f049b8884f6b34bdd1c754b113af19c280b9a33df7d1552395692064"
Jul 01 17:19:35.809068 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:35.809063    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ce016-predictor-6ffdb68b79-hq49w"
Jul 01 17:19:35.817610 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:35.817573    2563 generic.go:356] "Generic (PLEG): container finished" podID="318c4e47-af61-4745-be04-a6ca2637ed1e" containerID="57712d8baa361854be1ce1ba6abc303567ea7897a4316e2078a49c767c3b38ef" exitCode=0
Jul 01 17:19:35.817727 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:35.817631    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ce016-predictor-84dbff7cdb-rdb8n" event={"ID":"318c4e47-af61-4745-be04-a6ca2637ed1e","Type":"ContainerDied","Data":"57712d8baa361854be1ce1ba6abc303567ea7897a4316e2078a49c767c3b38ef"}
Jul 01 17:19:35.817727 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:35.817686    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ce016-predictor-84dbff7cdb-rdb8n" event={"ID":"318c4e47-af61-4745-be04-a6ca2637ed1e","Type":"ContainerDied","Data":"a9695844ccc1a8fd8098ac5eaa7c066019c685d25471e98e74defa80e0cbc5bc"}
Jul 01 17:19:35.817838 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:35.817774    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-ce016-predictor-84dbff7cdb-rdb8n"
Jul 01 17:19:35.823505 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:35.823484    2563 scope.go:117] "RemoveContainer" containerID="8973fa6bb27fafb35d84a407e0c752d080de1f6fb7fe9fb4c5e7804030c8a1d9"
Jul 01 17:19:35.831339 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:35.831321    2563 scope.go:117] "RemoveContainer" containerID="c391b181f049b8884f6b34bdd1c754b113af19c280b9a33df7d1552395692064"
Jul 01 17:19:35.831604 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:19:35.831584    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c391b181f049b8884f6b34bdd1c754b113af19c280b9a33df7d1552395692064\": container with ID starting with c391b181f049b8884f6b34bdd1c754b113af19c280b9a33df7d1552395692064 not found: ID does not exist" containerID="c391b181f049b8884f6b34bdd1c754b113af19c280b9a33df7d1552395692064"
Jul 01 17:19:35.831668 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:35.831612    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c391b181f049b8884f6b34bdd1c754b113af19c280b9a33df7d1552395692064"} err="failed to get container status \"c391b181f049b8884f6b34bdd1c754b113af19c280b9a33df7d1552395692064\": rpc error: code = NotFound desc = could not find container \"c391b181f049b8884f6b34bdd1c754b113af19c280b9a33df7d1552395692064\": container with ID starting with c391b181f049b8884f6b34bdd1c754b113af19c280b9a33df7d1552395692064 not found: ID does not exist"
Jul 01 17:19:35.831668 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:35.831630    2563 scope.go:117] "RemoveContainer" containerID="8973fa6bb27fafb35d84a407e0c752d080de1f6fb7fe9fb4c5e7804030c8a1d9"
Jul 01 17:19:35.831859 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:19:35.831845    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8973fa6bb27fafb35d84a407e0c752d080de1f6fb7fe9fb4c5e7804030c8a1d9\": container with ID starting with 8973fa6bb27fafb35d84a407e0c752d080de1f6fb7fe9fb4c5e7804030c8a1d9 not found: ID does not exist" containerID="8973fa6bb27fafb35d84a407e0c752d080de1f6fb7fe9fb4c5e7804030c8a1d9"
Jul 01 17:19:35.831896 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:35.831862    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8973fa6bb27fafb35d84a407e0c752d080de1f6fb7fe9fb4c5e7804030c8a1d9"} err="failed to get container status \"8973fa6bb27fafb35d84a407e0c752d080de1f6fb7fe9fb4c5e7804030c8a1d9\": rpc error: code = NotFound desc = could not find container \"8973fa6bb27fafb35d84a407e0c752d080de1f6fb7fe9fb4c5e7804030c8a1d9\": container with ID starting with 8973fa6bb27fafb35d84a407e0c752d080de1f6fb7fe9fb4c5e7804030c8a1d9 not found: ID does not exist"
Jul 01 17:19:35.831896 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:35.831874    2563 scope.go:117] "RemoveContainer" containerID="ccdf69b4fd13e204738793a3890ee4b0ed6e8d50d5334d772710f529dab71379"
Jul 01 17:19:35.839159 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:35.839135    2563 scope.go:117] "RemoveContainer" containerID="57712d8baa361854be1ce1ba6abc303567ea7897a4316e2078a49c767c3b38ef"
Jul 01 17:19:35.842575 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:35.842555    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ce016-predictor-6ffdb68b79-hq49w"]
Jul 01 17:19:35.846831 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:35.846807    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ce016-predictor-6ffdb68b79-hq49w"]
Jul 01 17:19:35.847184 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:35.847167    2563 scope.go:117] "RemoveContainer" containerID="ccdf69b4fd13e204738793a3890ee4b0ed6e8d50d5334d772710f529dab71379"
Jul 01 17:19:35.847488 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:19:35.847469    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccdf69b4fd13e204738793a3890ee4b0ed6e8d50d5334d772710f529dab71379\": container with ID starting with ccdf69b4fd13e204738793a3890ee4b0ed6e8d50d5334d772710f529dab71379 not found: ID does not exist" containerID="ccdf69b4fd13e204738793a3890ee4b0ed6e8d50d5334d772710f529dab71379"
Jul 01 17:19:35.847561 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:35.847498    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccdf69b4fd13e204738793a3890ee4b0ed6e8d50d5334d772710f529dab71379"} err="failed to get container status \"ccdf69b4fd13e204738793a3890ee4b0ed6e8d50d5334d772710f529dab71379\": rpc error: code = NotFound desc = could not find container \"ccdf69b4fd13e204738793a3890ee4b0ed6e8d50d5334d772710f529dab71379\": container with ID starting with ccdf69b4fd13e204738793a3890ee4b0ed6e8d50d5334d772710f529dab71379 not found: ID does not exist"
Jul 01 17:19:35.847561 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:35.847521    2563 scope.go:117] "RemoveContainer" containerID="57712d8baa361854be1ce1ba6abc303567ea7897a4316e2078a49c767c3b38ef"
Jul 01 17:19:35.847782 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:19:35.847763    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57712d8baa361854be1ce1ba6abc303567ea7897a4316e2078a49c767c3b38ef\": container with ID starting with 57712d8baa361854be1ce1ba6abc303567ea7897a4316e2078a49c767c3b38ef not found: ID does not exist" containerID="57712d8baa361854be1ce1ba6abc303567ea7897a4316e2078a49c767c3b38ef"
Jul 01 17:19:35.847838 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:35.847789    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57712d8baa361854be1ce1ba6abc303567ea7897a4316e2078a49c767c3b38ef"} err="failed to get container status \"57712d8baa361854be1ce1ba6abc303567ea7897a4316e2078a49c767c3b38ef\": rpc error: code = NotFound desc = could not find container \"57712d8baa361854be1ce1ba6abc303567ea7897a4316e2078a49c767c3b38ef\": container with ID starting with 57712d8baa361854be1ce1ba6abc303567ea7897a4316e2078a49c767c3b38ef not found: ID does not exist"
Jul 01 17:19:35.873075 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:35.873054    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ce016-predictor-84dbff7cdb-rdb8n"]
Jul 01 17:19:35.878782 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:35.878764    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ce016-predictor-84dbff7cdb-rdb8n"]
Jul 01 17:19:36.697726 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:36.697648    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="318c4e47-af61-4745-be04-a6ca2637ed1e" path="/var/lib/kubelet/pods/318c4e47-af61-4745-be04-a6ca2637ed1e/volumes"
Jul 01 17:19:36.698062 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:36.698049    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bee5b713-8b32-437d-86ad-c6c646b805c3" path="/var/lib/kubelet/pods/bee5b713-8b32-437d-86ad-c6c646b805c3/volumes"
Jul 01 17:19:38.803165 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:38.803136    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-1f194-predictor-6dd685749f-bq5w2"
Jul 01 17:19:38.803572 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:38.803524    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-1f194-predictor-6c5d4fc46b-kk5sz"
Jul 01 17:19:38.803705 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:38.803680    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1f194-predictor-6dd685749f-bq5w2" podUID="acb61295-6868-4bc3-9a75-77fae13527fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 01 17:19:38.803906 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:38.803883    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1f194-predictor-6c5d4fc46b-kk5sz" podUID="dc2febf1-bc84-4180-8f3c-da46de283e1b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 01 17:19:43.727122 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:43.727074    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-52e43-predictor-79768fc67d-xk27k" podUID="e49336bc-9319-467b-a6b8-969be03815e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 01 17:19:44.735113 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:44.735074    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-52e43-predictor-674cd6cc9-prtkw" podUID="430b61ee-095e-470f-a487-7d790277b37d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 01 17:19:48.804097 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:48.804056    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1f194-predictor-6dd685749f-bq5w2" podUID="acb61295-6868-4bc3-9a75-77fae13527fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 01 17:19:48.804602 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:48.804064    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1f194-predictor-6c5d4fc46b-kk5sz" podUID="dc2febf1-bc84-4180-8f3c-da46de283e1b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 01 17:19:53.727131 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:53.727083    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-52e43-predictor-79768fc67d-xk27k" podUID="e49336bc-9319-467b-a6b8-969be03815e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 01 17:19:54.734442 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:54.734399    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-52e43-predictor-674cd6cc9-prtkw" podUID="430b61ee-095e-470f-a487-7d790277b37d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 01 17:19:58.804625 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:58.804588    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1f194-predictor-6c5d4fc46b-kk5sz" podUID="dc2febf1-bc84-4180-8f3c-da46de283e1b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 01 17:19:58.805095 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:19:58.804590    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1f194-predictor-6dd685749f-bq5w2" podUID="acb61295-6868-4bc3-9a75-77fae13527fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 01 17:20:02.487925 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:02.487888    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:20:02.548843 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:02.548810    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:20:03.727014 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:03.726975    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-52e43-predictor-79768fc67d-xk27k" podUID="e49336bc-9319-467b-a6b8-969be03815e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 01 17:20:04.735396 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:04.735366    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-52e43-predictor-674cd6cc9-prtkw"
Jul 01 17:20:08.804558 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:08.804508    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1f194-predictor-6dd685749f-bq5w2" podUID="acb61295-6868-4bc3-9a75-77fae13527fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 01 17:20:08.804946 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:08.804508    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1f194-predictor-6c5d4fc46b-kk5sz" podUID="dc2febf1-bc84-4180-8f3c-da46de283e1b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 01 17:20:13.727526 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:13.727495    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-52e43-predictor-79768fc67d-xk27k"
Jul 01 17:20:15.563236 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:15.563208    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:20:15.656228 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:15.656197    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:20:18.804361 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:18.804308    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1f194-predictor-6c5d4fc46b-kk5sz" podUID="dc2febf1-bc84-4180-8f3c-da46de283e1b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 01 17:20:18.804744 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:18.804308    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1f194-predictor-6dd685749f-bq5w2" podUID="acb61295-6868-4bc3-9a75-77fae13527fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 01 17:20:25.650726 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:25.650643    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:20:25.765990 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:25.765959    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-52e43-predictor-674cd6cc9-prtkw"]
Jul 01 17:20:25.766239 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:25.766190    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-52e43-predictor-674cd6cc9-prtkw" podUID="430b61ee-095e-470f-a487-7d790277b37d" containerName="kserve-container" containerID="cri-o://9c4643a1bdda6c42cd25206ebaee613f1b0f93ea2d9d1becea3c9dc259344dc7" gracePeriod=30
Jul 01 17:20:25.766338 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:25.766239    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-52e43-predictor-674cd6cc9-prtkw" podUID="430b61ee-095e-470f-a487-7d790277b37d" containerName="kube-rbac-proxy" containerID="cri-o://2cd75215f5ea925dd20a078b5135506896d83f8e825b469b6b0117b3d09edbb3" gracePeriod=30
Jul 01 17:20:25.793007 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:25.792980    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bbcac-predictor-7874fb45b7-bmwwk"]
Jul 01 17:20:25.793321 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:25.793308    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="318c4e47-af61-4745-be04-a6ca2637ed1e" containerName="kube-rbac-proxy"
Jul 01 17:20:25.793389 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:25.793323    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="318c4e47-af61-4745-be04-a6ca2637ed1e" containerName="kube-rbac-proxy"
Jul 01 17:20:25.793389 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:25.793339    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="318c4e47-af61-4745-be04-a6ca2637ed1e" containerName="kserve-container"
Jul 01 17:20:25.793389 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:25.793362    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="318c4e47-af61-4745-be04-a6ca2637ed1e" containerName="kserve-container"
Jul 01 17:20:25.793389 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:25.793368    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bee5b713-8b32-437d-86ad-c6c646b805c3" containerName="kserve-container"
Jul 01 17:20:25.793389 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:25.793374    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="bee5b713-8b32-437d-86ad-c6c646b805c3" containerName="kserve-container"
Jul 01 17:20:25.793389 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:25.793381    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bee5b713-8b32-437d-86ad-c6c646b805c3" containerName="kube-rbac-proxy"
Jul 01 17:20:25.793389 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:25.793386    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="bee5b713-8b32-437d-86ad-c6c646b805c3" containerName="kube-rbac-proxy"
Jul 01 17:20:25.793598 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:25.793440    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="318c4e47-af61-4745-be04-a6ca2637ed1e" containerName="kserve-container"
Jul 01 17:20:25.793598 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:25.793448    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="318c4e47-af61-4745-be04-a6ca2637ed1e" containerName="kube-rbac-proxy"
Jul 01 17:20:25.793598 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:25.793455    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="bee5b713-8b32-437d-86ad-c6c646b805c3" containerName="kube-rbac-proxy"
Jul 01 17:20:25.793598 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:25.793461    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="bee5b713-8b32-437d-86ad-c6c646b805c3" containerName="kserve-container"
Jul 01 17:20:25.797920 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:25.797900    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-bbcac-predictor-7874fb45b7-bmwwk"
Jul 01 17:20:25.800676 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:25.800657    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-bbcac-kube-rbac-proxy-sar-config\""
Jul 01 17:20:25.800776 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:25.800664    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-bbcac-predictor-serving-cert\""
Jul 01 17:20:25.810422 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:25.808567    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bbcac-predictor-7874fb45b7-bmwwk"]
Jul 01 17:20:25.910427 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:25.908742    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19df0b11-647e-4826-99c7-a3016979c15b-proxy-tls\") pod \"success-200-isvc-bbcac-predictor-7874fb45b7-bmwwk\" (UID: \"19df0b11-647e-4826-99c7-a3016979c15b\") " pod="kserve-ci-e2e-test/success-200-isvc-bbcac-predictor-7874fb45b7-bmwwk"
Jul 01 17:20:25.910427 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:25.908817    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztwrx\" (UniqueName: \"kubernetes.io/projected/19df0b11-647e-4826-99c7-a3016979c15b-kube-api-access-ztwrx\") pod \"success-200-isvc-bbcac-predictor-7874fb45b7-bmwwk\" (UID: \"19df0b11-647e-4826-99c7-a3016979c15b\") " pod="kserve-ci-e2e-test/success-200-isvc-bbcac-predictor-7874fb45b7-bmwwk"
Jul 01 17:20:25.910427 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:25.908876    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19df0b11-647e-4826-99c7-a3016979c15b-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-bbcac-predictor-7874fb45b7-bmwwk\" (UID: \"19df0b11-647e-4826-99c7-a3016979c15b\") " pod="kserve-ci-e2e-test/success-200-isvc-bbcac-predictor-7874fb45b7-bmwwk"
Jul 01 17:20:25.913376 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:25.911757    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bbcac-predictor-5ff4fd5cc4-wtk7h"]
Jul 01 17:20:25.917784 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:25.917749    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:20:25.919840 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:25.919806    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bbcac-predictor-5ff4fd5cc4-wtk7h"]
Jul 01 17:20:25.919976 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:25.919952    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-bbcac-predictor-5ff4fd5cc4-wtk7h"
Jul 01 17:20:25.922831 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:25.922811    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-bbcac-predictor-serving-cert\""
Jul 01 17:20:25.922927 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:25.922812    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-bbcac-kube-rbac-proxy-sar-config\""
Jul 01 17:20:25.959564 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:25.959537    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-52e43-predictor-79768fc67d-xk27k"]
Jul 01 17:20:25.959793 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:25.959762    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-52e43-predictor-79768fc67d-xk27k" podUID="e49336bc-9319-467b-a6b8-969be03815e0" containerName="kserve-container" containerID="cri-o://e4e01e2a3a4246ebfe8d9a9aa6defbdddb1049c0553b307828c3677bd661ddc0" gracePeriod=30
Jul 01 17:20:25.959974 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:25.959950    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-52e43-predictor-79768fc67d-xk27k" podUID="e49336bc-9319-467b-a6b8-969be03815e0" containerName="kube-rbac-proxy" containerID="cri-o://b67ca1549736c9d0c079978edcbf7e352faae97b5d7de801fb52049174d900b9" gracePeriod=30
Jul 01 17:20:26.009708 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:26.009660    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b4cccca-1a19-448a-b584-acf00f272af2-proxy-tls\") pod \"error-404-isvc-bbcac-predictor-5ff4fd5cc4-wtk7h\" (UID: \"9b4cccca-1a19-448a-b584-acf00f272af2\") " pod="kserve-ci-e2e-test/error-404-isvc-bbcac-predictor-5ff4fd5cc4-wtk7h"
Jul 01 17:20:26.009708 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:26.009710    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b4cccca-1a19-448a-b584-acf00f272af2-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-bbcac-predictor-5ff4fd5cc4-wtk7h\" (UID: \"9b4cccca-1a19-448a-b584-acf00f272af2\") " pod="kserve-ci-e2e-test/error-404-isvc-bbcac-predictor-5ff4fd5cc4-wtk7h"
Jul 01 17:20:26.009938 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:26.009798    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19df0b11-647e-4826-99c7-a3016979c15b-proxy-tls\") pod \"success-200-isvc-bbcac-predictor-7874fb45b7-bmwwk\" (UID: \"19df0b11-647e-4826-99c7-a3016979c15b\") " pod="kserve-ci-e2e-test/success-200-isvc-bbcac-predictor-7874fb45b7-bmwwk"
Jul 01 17:20:26.009938 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:26.009836    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztwrx\" (UniqueName: \"kubernetes.io/projected/19df0b11-647e-4826-99c7-a3016979c15b-kube-api-access-ztwrx\") pod \"success-200-isvc-bbcac-predictor-7874fb45b7-bmwwk\" (UID: \"19df0b11-647e-4826-99c7-a3016979c15b\") " pod="kserve-ci-e2e-test/success-200-isvc-bbcac-predictor-7874fb45b7-bmwwk"
Jul 01 17:20:26.009938 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:26.009864    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m6mt\" (UniqueName: \"kubernetes.io/projected/9b4cccca-1a19-448a-b584-acf00f272af2-kube-api-access-9m6mt\") pod \"error-404-isvc-bbcac-predictor-5ff4fd5cc4-wtk7h\" (UID: \"9b4cccca-1a19-448a-b584-acf00f272af2\") " pod="kserve-ci-e2e-test/error-404-isvc-bbcac-predictor-5ff4fd5cc4-wtk7h"
Jul 01 17:20:26.009938 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:26.009920    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19df0b11-647e-4826-99c7-a3016979c15b-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-bbcac-predictor-7874fb45b7-bmwwk\" (UID: \"19df0b11-647e-4826-99c7-a3016979c15b\") " pod="kserve-ci-e2e-test/success-200-isvc-bbcac-predictor-7874fb45b7-bmwwk"
Jul 01 17:20:26.010690 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:26.010654    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19df0b11-647e-4826-99c7-a3016979c15b-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-bbcac-predictor-7874fb45b7-bmwwk\" (UID: \"19df0b11-647e-4826-99c7-a3016979c15b\") " pod="kserve-ci-e2e-test/success-200-isvc-bbcac-predictor-7874fb45b7-bmwwk"
Jul 01 17:20:26.012806 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:26.012775    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19df0b11-647e-4826-99c7-a3016979c15b-proxy-tls\") pod \"success-200-isvc-bbcac-predictor-7874fb45b7-bmwwk\" (UID: \"19df0b11-647e-4826-99c7-a3016979c15b\") " pod="kserve-ci-e2e-test/success-200-isvc-bbcac-predictor-7874fb45b7-bmwwk"
Jul 01 17:20:26.016260 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:26.016235    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:20:26.019447 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:26.019427    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztwrx\" (UniqueName: \"kubernetes.io/projected/19df0b11-647e-4826-99c7-a3016979c15b-kube-api-access-ztwrx\") pod \"success-200-isvc-bbcac-predictor-7874fb45b7-bmwwk\" (UID: \"19df0b11-647e-4826-99c7-a3016979c15b\") " pod="kserve-ci-e2e-test/success-200-isvc-bbcac-predictor-7874fb45b7-bmwwk"
Jul 01 17:20:26.033196 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:26.033169    2563 generic.go:356] "Generic (PLEG): container finished" podID="430b61ee-095e-470f-a487-7d790277b37d" containerID="2cd75215f5ea925dd20a078b5135506896d83f8e825b469b6b0117b3d09edbb3" exitCode=2
Jul 01 17:20:26.033328 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:26.033247    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-52e43-predictor-674cd6cc9-prtkw" event={"ID":"430b61ee-095e-470f-a487-7d790277b37d","Type":"ContainerDied","Data":"2cd75215f5ea925dd20a078b5135506896d83f8e825b469b6b0117b3d09edbb3"}
Jul 01 17:20:26.111446 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:26.111401    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b4cccca-1a19-448a-b584-acf00f272af2-proxy-tls\") pod \"error-404-isvc-bbcac-predictor-5ff4fd5cc4-wtk7h\" (UID: \"9b4cccca-1a19-448a-b584-acf00f272af2\") " pod="kserve-ci-e2e-test/error-404-isvc-bbcac-predictor-5ff4fd5cc4-wtk7h"
Jul 01 17:20:26.111446 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:26.111449    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b4cccca-1a19-448a-b584-acf00f272af2-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-bbcac-predictor-5ff4fd5cc4-wtk7h\" (UID: \"9b4cccca-1a19-448a-b584-acf00f272af2\") " pod="kserve-ci-e2e-test/error-404-isvc-bbcac-predictor-5ff4fd5cc4-wtk7h"
Jul 01 17:20:26.111716 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:26.111499    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m6mt\" (UniqueName: \"kubernetes.io/projected/9b4cccca-1a19-448a-b584-acf00f272af2-kube-api-access-9m6mt\") pod \"error-404-isvc-bbcac-predictor-5ff4fd5cc4-wtk7h\" (UID: \"9b4cccca-1a19-448a-b584-acf00f272af2\") " pod="kserve-ci-e2e-test/error-404-isvc-bbcac-predictor-5ff4fd5cc4-wtk7h"
Jul 01 17:20:26.111716 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:20:26.111562    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-bbcac-predictor-serving-cert: secret "error-404-isvc-bbcac-predictor-serving-cert" not found
Jul 01 17:20:26.111716 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:20:26.111642    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b4cccca-1a19-448a-b584-acf00f272af2-proxy-tls podName:9b4cccca-1a19-448a-b584-acf00f272af2 nodeName:}" failed. No retries permitted until 2026-07-01 17:20:26.611620104 +0000 UTC m=+1344.408758496 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9b4cccca-1a19-448a-b584-acf00f272af2-proxy-tls") pod "error-404-isvc-bbcac-predictor-5ff4fd5cc4-wtk7h" (UID: "9b4cccca-1a19-448a-b584-acf00f272af2") : secret "error-404-isvc-bbcac-predictor-serving-cert" not found
Jul 01 17:20:26.112135 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:26.112106    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b4cccca-1a19-448a-b584-acf00f272af2-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-bbcac-predictor-5ff4fd5cc4-wtk7h\" (UID: \"9b4cccca-1a19-448a-b584-acf00f272af2\") " pod="kserve-ci-e2e-test/error-404-isvc-bbcac-predictor-5ff4fd5cc4-wtk7h"
Jul 01 17:20:26.113568 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:26.113542    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-bbcac-predictor-7874fb45b7-bmwwk"
Jul 01 17:20:26.121453 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:26.121423    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m6mt\" (UniqueName: \"kubernetes.io/projected/9b4cccca-1a19-448a-b584-acf00f272af2-kube-api-access-9m6mt\") pod \"error-404-isvc-bbcac-predictor-5ff4fd5cc4-wtk7h\" (UID: \"9b4cccca-1a19-448a-b584-acf00f272af2\") " pod="kserve-ci-e2e-test/error-404-isvc-bbcac-predictor-5ff4fd5cc4-wtk7h"
Jul 01 17:20:26.259364 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:26.259297    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:20:26.263914 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:26.263840    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bbcac-predictor-7874fb45b7-bmwwk"]
Jul 01 17:20:26.266312 ip-10-0-137-180 kubenswrapper[2563]: W0701 17:20:26.266285    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19df0b11_647e_4826_99c7_a3016979c15b.slice/crio-116c82a114b623aade2c6af9195d7295680741e88db055f7ca92993724647137 WatchSource:0}: Error finding container 116c82a114b623aade2c6af9195d7295680741e88db055f7ca92993724647137: Status 404 returned error can't find the container with id 116c82a114b623aade2c6af9195d7295680741e88db055f7ca92993724647137
Jul 01 17:20:26.616685 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:26.616639    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b4cccca-1a19-448a-b584-acf00f272af2-proxy-tls\") pod \"error-404-isvc-bbcac-predictor-5ff4fd5cc4-wtk7h\" (UID: \"9b4cccca-1a19-448a-b584-acf00f272af2\") " pod="kserve-ci-e2e-test/error-404-isvc-bbcac-predictor-5ff4fd5cc4-wtk7h"
Jul 01 17:20:26.619206 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:26.619178    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b4cccca-1a19-448a-b584-acf00f272af2-proxy-tls\") pod \"error-404-isvc-bbcac-predictor-5ff4fd5cc4-wtk7h\" (UID: \"9b4cccca-1a19-448a-b584-acf00f272af2\") " pod="kserve-ci-e2e-test/error-404-isvc-bbcac-predictor-5ff4fd5cc4-wtk7h"
Jul 01 17:20:26.830692 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:26.830643    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-bbcac-predictor-5ff4fd5cc4-wtk7h"
Jul 01 17:20:26.961015 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:26.960990    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bbcac-predictor-5ff4fd5cc4-wtk7h"]
Jul 01 17:20:26.963329 ip-10-0-137-180 kubenswrapper[2563]: W0701 17:20:26.963299    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b4cccca_1a19_448a_b584_acf00f272af2.slice/crio-c70deaf1d76e862a2ec16e0ca31a602442cccfac4bcfe87c9053ed9845e1fea3 WatchSource:0}: Error finding container c70deaf1d76e862a2ec16e0ca31a602442cccfac4bcfe87c9053ed9845e1fea3: Status 404 returned error can't find the container with id c70deaf1d76e862a2ec16e0ca31a602442cccfac4bcfe87c9053ed9845e1fea3
Jul 01 17:20:27.039810 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:27.039783    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bbcac-predictor-5ff4fd5cc4-wtk7h" event={"ID":"9b4cccca-1a19-448a-b584-acf00f272af2","Type":"ContainerStarted","Data":"d0aa041de1e5bdae243c0daeb941e569c5bd4c7e1a1ce982cacee42828cd25bf"}
Jul 01 17:20:27.039919 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:27.039846    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bbcac-predictor-5ff4fd5cc4-wtk7h" event={"ID":"9b4cccca-1a19-448a-b584-acf00f272af2","Type":"ContainerStarted","Data":"c70deaf1d76e862a2ec16e0ca31a602442cccfac4bcfe87c9053ed9845e1fea3"}
Jul 01 17:20:27.041883 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:27.041853    2563 generic.go:356] "Generic (PLEG): container finished" podID="e49336bc-9319-467b-a6b8-969be03815e0" containerID="b67ca1549736c9d0c079978edcbf7e352faae97b5d7de801fb52049174d900b9" exitCode=2
Jul 01 17:20:27.041963 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:27.041920    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-52e43-predictor-79768fc67d-xk27k" event={"ID":"e49336bc-9319-467b-a6b8-969be03815e0","Type":"ContainerDied","Data":"b67ca1549736c9d0c079978edcbf7e352faae97b5d7de801fb52049174d900b9"}
Jul 01 17:20:27.043480 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:27.043462    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bbcac-predictor-7874fb45b7-bmwwk" event={"ID":"19df0b11-647e-4826-99c7-a3016979c15b","Type":"ContainerStarted","Data":"72652a651bf57110bb580b3cf4a116557e165c3296299f04d84b893f76d6ecd2"}
Jul 01 17:20:27.043557 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:27.043510    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bbcac-predictor-7874fb45b7-bmwwk" event={"ID":"19df0b11-647e-4826-99c7-a3016979c15b","Type":"ContainerStarted","Data":"f9a9355f125fe56738b38a716843c3e98d4dba400d4e5ec94283f93bec92ea19"}
Jul 01 17:20:27.043557 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:27.043536    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bbcac-predictor-7874fb45b7-bmwwk" event={"ID":"19df0b11-647e-4826-99c7-a3016979c15b","Type":"ContainerStarted","Data":"116c82a114b623aade2c6af9195d7295680741e88db055f7ca92993724647137"}
Jul 01 17:20:27.043625 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:27.043558    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-bbcac-predictor-7874fb45b7-bmwwk"
Jul 01 17:20:27.043625 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:27.043588    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-bbcac-predictor-7874fb45b7-bmwwk"
Jul 01 17:20:27.045107 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:27.045085    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bbcac-predictor-7874fb45b7-bmwwk" podUID="19df0b11-647e-4826-99c7-a3016979c15b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 01 17:20:27.064604 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:27.064562    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-bbcac-predictor-7874fb45b7-bmwwk" podStartSLOduration=2.064545344 podStartE2EDuration="2.064545344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:20:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:20:27.063339319 +0000 UTC m=+1344.860477739" watchObservedRunningTime="2026-07-01 17:20:27.064545344 +0000 UTC m=+1344.861683744"
Jul 01 17:20:28.049725 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:28.049685    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bbcac-predictor-5ff4fd5cc4-wtk7h" event={"ID":"9b4cccca-1a19-448a-b584-acf00f272af2","Type":"ContainerStarted","Data":"f9177a7b1605443abf5df136ab7e4cdaf04b1d4e62021480e8761e7387b34717"}
Jul 01 17:20:28.050164 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:28.050068    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-bbcac-predictor-5ff4fd5cc4-wtk7h"
Jul 01 17:20:28.050164 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:28.050116    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-bbcac-predictor-5ff4fd5cc4-wtk7h"
Jul 01 17:20:28.050164 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:28.050132    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bbcac-predictor-7874fb45b7-bmwwk" podUID="19df0b11-647e-4826-99c7-a3016979c15b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 01 17:20:28.051523 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:28.051496    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bbcac-predictor-5ff4fd5cc4-wtk7h" podUID="9b4cccca-1a19-448a-b584-acf00f272af2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 01 17:20:28.069968 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:28.069923    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-bbcac-predictor-5ff4fd5cc4-wtk7h" podStartSLOduration=3.069909456 podStartE2EDuration="3.069909456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:20:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:20:28.068653443 +0000 UTC m=+1345.865791856" watchObservedRunningTime="2026-07-01 17:20:28.069909456 +0000 UTC m=+1345.867047896"
Jul 01 17:20:28.721103 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:28.721063    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-52e43-predictor-79768fc67d-xk27k" podUID="e49336bc-9319-467b-a6b8-969be03815e0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused"
Jul 01 17:20:28.805014 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:28.804985    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-1f194-predictor-6dd685749f-bq5w2"
Jul 01 17:20:28.805532 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:28.805510    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-1f194-predictor-6c5d4fc46b-kk5sz"
Jul 01 17:20:29.059475 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:29.059153    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bbcac-predictor-5ff4fd5cc4-wtk7h" podUID="9b4cccca-1a19-448a-b584-acf00f272af2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 01 17:20:29.409226 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:29.409196    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-52e43-predictor-79768fc67d-xk27k"
Jul 01 17:20:29.546462 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:29.546434    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e49336bc-9319-467b-a6b8-969be03815e0-kube-rbac-proxy-sar-config\") pod \"e49336bc-9319-467b-a6b8-969be03815e0\" (UID: \"e49336bc-9319-467b-a6b8-969be03815e0\") "
Jul 01 17:20:29.546601 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:29.546491    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e49336bc-9319-467b-a6b8-969be03815e0-proxy-tls\") pod \"e49336bc-9319-467b-a6b8-969be03815e0\" (UID: \"e49336bc-9319-467b-a6b8-969be03815e0\") "
Jul 01 17:20:29.546601 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:29.546520    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s2ctl\" (UniqueName: \"kubernetes.io/projected/e49336bc-9319-467b-a6b8-969be03815e0-kube-api-access-s2ctl\") pod \"e49336bc-9319-467b-a6b8-969be03815e0\" (UID: \"e49336bc-9319-467b-a6b8-969be03815e0\") "
Jul 01 17:20:29.546770 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:29.546744    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e49336bc-9319-467b-a6b8-969be03815e0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e49336bc-9319-467b-a6b8-969be03815e0" (UID: "e49336bc-9319-467b-a6b8-969be03815e0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:20:29.548596 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:29.548575    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e49336bc-9319-467b-a6b8-969be03815e0-kube-api-access-s2ctl" (OuterVolumeSpecName: "kube-api-access-s2ctl") pod "e49336bc-9319-467b-a6b8-969be03815e0" (UID: "e49336bc-9319-467b-a6b8-969be03815e0"). InnerVolumeSpecName "kube-api-access-s2ctl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 17:20:29.548664 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:29.548600    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e49336bc-9319-467b-a6b8-969be03815e0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e49336bc-9319-467b-a6b8-969be03815e0" (UID: "e49336bc-9319-467b-a6b8-969be03815e0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 17:20:29.647996 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:29.647967    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e49336bc-9319-467b-a6b8-969be03815e0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:20:29.648118 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:29.648000    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e49336bc-9319-467b-a6b8-969be03815e0-proxy-tls\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:20:29.648118 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:29.648016    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2ctl\" (UniqueName: \"kubernetes.io/projected/e49336bc-9319-467b-a6b8-969be03815e0-kube-api-access-s2ctl\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:20:29.702207 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:29.702182    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-52e43-predictor-674cd6cc9-prtkw"
Jul 01 17:20:29.749166 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:29.749133    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/430b61ee-095e-470f-a487-7d790277b37d-proxy-tls\") pod \"430b61ee-095e-470f-a487-7d790277b37d\" (UID: \"430b61ee-095e-470f-a487-7d790277b37d\") "
Jul 01 17:20:29.749329 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:29.749183    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8spwr\" (UniqueName: \"kubernetes.io/projected/430b61ee-095e-470f-a487-7d790277b37d-kube-api-access-8spwr\") pod \"430b61ee-095e-470f-a487-7d790277b37d\" (UID: \"430b61ee-095e-470f-a487-7d790277b37d\") "
Jul 01 17:20:29.749329 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:29.749210    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/430b61ee-095e-470f-a487-7d790277b37d-kube-rbac-proxy-sar-config\") pod \"430b61ee-095e-470f-a487-7d790277b37d\" (UID: \"430b61ee-095e-470f-a487-7d790277b37d\") "
Jul 01 17:20:29.749608 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:29.749581    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/430b61ee-095e-470f-a487-7d790277b37d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "430b61ee-095e-470f-a487-7d790277b37d" (UID: "430b61ee-095e-470f-a487-7d790277b37d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:20:29.751446 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:29.751412    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/430b61ee-095e-470f-a487-7d790277b37d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "430b61ee-095e-470f-a487-7d790277b37d" (UID: "430b61ee-095e-470f-a487-7d790277b37d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 17:20:29.751544 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:29.751491    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/430b61ee-095e-470f-a487-7d790277b37d-kube-api-access-8spwr" (OuterVolumeSpecName: "kube-api-access-8spwr") pod "430b61ee-095e-470f-a487-7d790277b37d" (UID: "430b61ee-095e-470f-a487-7d790277b37d"). InnerVolumeSpecName "kube-api-access-8spwr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 17:20:29.850360 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:29.850259    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/430b61ee-095e-470f-a487-7d790277b37d-proxy-tls\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:20:29.850360 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:29.850291    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8spwr\" (UniqueName: \"kubernetes.io/projected/430b61ee-095e-470f-a487-7d790277b37d-kube-api-access-8spwr\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:20:29.850360 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:29.850303    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/430b61ee-095e-470f-a487-7d790277b37d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:20:30.061551 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:30.061505    2563 generic.go:356] "Generic (PLEG): container finished" podID="e49336bc-9319-467b-a6b8-969be03815e0" containerID="e4e01e2a3a4246ebfe8d9a9aa6defbdddb1049c0553b307828c3677bd661ddc0" exitCode=0
Jul 01 17:20:30.061551 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:30.061553    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-52e43-predictor-79768fc67d-xk27k"
Jul 01 17:20:30.062123 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:30.061567    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-52e43-predictor-79768fc67d-xk27k" event={"ID":"e49336bc-9319-467b-a6b8-969be03815e0","Type":"ContainerDied","Data":"e4e01e2a3a4246ebfe8d9a9aa6defbdddb1049c0553b307828c3677bd661ddc0"}
Jul 01 17:20:30.062123 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:30.061621    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-52e43-predictor-79768fc67d-xk27k" event={"ID":"e49336bc-9319-467b-a6b8-969be03815e0","Type":"ContainerDied","Data":"12b5a2ab49208f7229b2ff7e0b66688f1586146fa1ac8305abfe97536e515cf0"}
Jul 01 17:20:30.062123 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:30.061655    2563 scope.go:117] "RemoveContainer" containerID="b67ca1549736c9d0c079978edcbf7e352faae97b5d7de801fb52049174d900b9"
Jul 01 17:20:30.063540 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:30.063511    2563 generic.go:356] "Generic (PLEG): container finished" podID="430b61ee-095e-470f-a487-7d790277b37d" containerID="9c4643a1bdda6c42cd25206ebaee613f1b0f93ea2d9d1becea3c9dc259344dc7" exitCode=0
Jul 01 17:20:30.063540 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:30.063537    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-52e43-predictor-674cd6cc9-prtkw" event={"ID":"430b61ee-095e-470f-a487-7d790277b37d","Type":"ContainerDied","Data":"9c4643a1bdda6c42cd25206ebaee613f1b0f93ea2d9d1becea3c9dc259344dc7"}
Jul 01 17:20:30.063758 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:30.063556    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-52e43-predictor-674cd6cc9-prtkw"
Jul 01 17:20:30.063758 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:30.063589    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-52e43-predictor-674cd6cc9-prtkw" event={"ID":"430b61ee-095e-470f-a487-7d790277b37d","Type":"ContainerDied","Data":"29703f53293de1a8ce91cec427afcda1e102f2a4eb75e24aa33249e583a500de"}
Jul 01 17:20:30.064157 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:30.064121    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bbcac-predictor-5ff4fd5cc4-wtk7h" podUID="9b4cccca-1a19-448a-b584-acf00f272af2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 01 17:20:30.071778 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:30.071763    2563 scope.go:117] "RemoveContainer" containerID="e4e01e2a3a4246ebfe8d9a9aa6defbdddb1049c0553b307828c3677bd661ddc0"
Jul 01 17:20:30.080090 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:30.080074    2563 scope.go:117] "RemoveContainer" containerID="b67ca1549736c9d0c079978edcbf7e352faae97b5d7de801fb52049174d900b9"
Jul 01 17:20:30.080325 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:20:30.080309    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b67ca1549736c9d0c079978edcbf7e352faae97b5d7de801fb52049174d900b9\": container with ID starting with b67ca1549736c9d0c079978edcbf7e352faae97b5d7de801fb52049174d900b9 not found: ID does not exist" containerID="b67ca1549736c9d0c079978edcbf7e352faae97b5d7de801fb52049174d900b9"
Jul 01 17:20:30.080387 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:30.080334    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b67ca1549736c9d0c079978edcbf7e352faae97b5d7de801fb52049174d900b9"} err="failed to get container status \"b67ca1549736c9d0c079978edcbf7e352faae97b5d7de801fb52049174d900b9\": rpc error: code = NotFound desc = could not find container \"b67ca1549736c9d0c079978edcbf7e352faae97b5d7de801fb52049174d900b9\": container with ID starting with b67ca1549736c9d0c079978edcbf7e352faae97b5d7de801fb52049174d900b9 not found: ID does not exist"
Jul 01 17:20:30.080387 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:30.080376    2563 scope.go:117] "RemoveContainer" containerID="e4e01e2a3a4246ebfe8d9a9aa6defbdddb1049c0553b307828c3677bd661ddc0"
Jul 01 17:20:30.080646 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:20:30.080631    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4e01e2a3a4246ebfe8d9a9aa6defbdddb1049c0553b307828c3677bd661ddc0\": container with ID starting with e4e01e2a3a4246ebfe8d9a9aa6defbdddb1049c0553b307828c3677bd661ddc0 not found: ID does not exist" containerID="e4e01e2a3a4246ebfe8d9a9aa6defbdddb1049c0553b307828c3677bd661ddc0"
Jul 01 17:20:30.080695 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:30.080650    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4e01e2a3a4246ebfe8d9a9aa6defbdddb1049c0553b307828c3677bd661ddc0"} err="failed to get container status \"e4e01e2a3a4246ebfe8d9a9aa6defbdddb1049c0553b307828c3677bd661ddc0\": rpc error: code = NotFound desc = could not find container \"e4e01e2a3a4246ebfe8d9a9aa6defbdddb1049c0553b307828c3677bd661ddc0\": container with ID starting with e4e01e2a3a4246ebfe8d9a9aa6defbdddb1049c0553b307828c3677bd661ddc0 not found: ID does not exist"
Jul 01 17:20:30.080695 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:30.080662    2563 scope.go:117] "RemoveContainer" containerID="2cd75215f5ea925dd20a078b5135506896d83f8e825b469b6b0117b3d09edbb3"
Jul 01 17:20:30.086522 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:30.086496    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-52e43-predictor-79768fc67d-xk27k"]
Jul 01 17:20:30.089072 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:30.089054    2563 scope.go:117] "RemoveContainer" containerID="9c4643a1bdda6c42cd25206ebaee613f1b0f93ea2d9d1becea3c9dc259344dc7"
Jul 01 17:20:30.089179 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:30.089160    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-52e43-predictor-79768fc67d-xk27k"]
Jul 01 17:20:30.098536 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:30.098513    2563 scope.go:117] "RemoveContainer" containerID="2cd75215f5ea925dd20a078b5135506896d83f8e825b469b6b0117b3d09edbb3"
Jul 01 17:20:30.098967 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:20:30.098946    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cd75215f5ea925dd20a078b5135506896d83f8e825b469b6b0117b3d09edbb3\": container with ID starting with 2cd75215f5ea925dd20a078b5135506896d83f8e825b469b6b0117b3d09edbb3 not found: ID does not exist" containerID="2cd75215f5ea925dd20a078b5135506896d83f8e825b469b6b0117b3d09edbb3"
Jul 01 17:20:30.099070 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:30.098973    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cd75215f5ea925dd20a078b5135506896d83f8e825b469b6b0117b3d09edbb3"} err="failed to get container status \"2cd75215f5ea925dd20a078b5135506896d83f8e825b469b6b0117b3d09edbb3\": rpc error: code = NotFound desc = could not find container \"2cd75215f5ea925dd20a078b5135506896d83f8e825b469b6b0117b3d09edbb3\": container with ID starting with 2cd75215f5ea925dd20a078b5135506896d83f8e825b469b6b0117b3d09edbb3 not found: ID does not exist"
Jul 01 17:20:30.099070 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:30.098989    2563 scope.go:117] "RemoveContainer" containerID="9c4643a1bdda6c42cd25206ebaee613f1b0f93ea2d9d1becea3c9dc259344dc7"
Jul 01 17:20:30.099239 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:20:30.099225    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c4643a1bdda6c42cd25206ebaee613f1b0f93ea2d9d1becea3c9dc259344dc7\": container with ID starting with 9c4643a1bdda6c42cd25206ebaee613f1b0f93ea2d9d1becea3c9dc259344dc7 not found: ID does not exist" containerID="9c4643a1bdda6c42cd25206ebaee613f1b0f93ea2d9d1becea3c9dc259344dc7"
Jul 01 17:20:30.099286 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:30.099242    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c4643a1bdda6c42cd25206ebaee613f1b0f93ea2d9d1becea3c9dc259344dc7"} err="failed to get container status \"9c4643a1bdda6c42cd25206ebaee613f1b0f93ea2d9d1becea3c9dc259344dc7\": rpc error: code = NotFound desc = could not find container \"9c4643a1bdda6c42cd25206ebaee613f1b0f93ea2d9d1becea3c9dc259344dc7\": container with ID starting with 9c4643a1bdda6c42cd25206ebaee613f1b0f93ea2d9d1becea3c9dc259344dc7 not found: ID does not exist"
Jul 01 17:20:30.100753 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:30.100711    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-52e43-predictor-674cd6cc9-prtkw"]
Jul 01 17:20:30.106131 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:30.106111    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-52e43-predictor-674cd6cc9-prtkw"]
Jul 01 17:20:30.697329 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:30.697295    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="430b61ee-095e-470f-a487-7d790277b37d" path="/var/lib/kubelet/pods/430b61ee-095e-470f-a487-7d790277b37d/volumes"
Jul 01 17:20:30.697723 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:30.697710    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e49336bc-9319-467b-a6b8-969be03815e0" path="/var/lib/kubelet/pods/e49336bc-9319-467b-a6b8-969be03815e0/volumes"
Jul 01 17:20:33.054942 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:33.054909    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-bbcac-predictor-7874fb45b7-bmwwk"
Jul 01 17:20:33.055485 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:33.055453    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bbcac-predictor-7874fb45b7-bmwwk" podUID="19df0b11-647e-4826-99c7-a3016979c15b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 01 17:20:35.069142 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:35.069113    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-bbcac-predictor-5ff4fd5cc4-wtk7h"
Jul 01 17:20:35.069594 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:35.069566    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bbcac-predictor-5ff4fd5cc4-wtk7h" podUID="9b4cccca-1a19-448a-b584-acf00f272af2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 01 17:20:41.406794 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:41.406753    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:20:41.740042 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:41.740006    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:20:43.056240 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:43.056203    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bbcac-predictor-7874fb45b7-bmwwk" podUID="19df0b11-647e-4826-99c7-a3016979c15b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 01 17:20:45.069782 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:45.069747    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bbcac-predictor-5ff4fd5cc4-wtk7h" podUID="9b4cccca-1a19-448a-b584-acf00f272af2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 01 17:20:51.486496 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:51.486464    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:20:51.552330 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:51.552297    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1f194-predictor-6c5d4fc46b-kk5sz"]
Jul 01 17:20:51.552872 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:51.552842    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-1f194-predictor-6c5d4fc46b-kk5sz" podUID="dc2febf1-bc84-4180-8f3c-da46de283e1b" containerName="kserve-container" containerID="cri-o://921fa68fcb3fcb73af90ff3ad0ee836e579e16f40df0df682dc03fc16d995ccd" gracePeriod=30
Jul 01 17:20:51.553085 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:51.552885    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-1f194-predictor-6c5d4fc46b-kk5sz" podUID="dc2febf1-bc84-4180-8f3c-da46de283e1b" containerName="kube-rbac-proxy" containerID="cri-o://34c308d4f10809a65bf679816d0f1b7e41d54324e6c09ed016a9236ee3d58cce" gracePeriod=30
Jul 01 17:20:51.596050 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:51.596009    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:20:51.618005 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:51.617971    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1f194-predictor-6dd685749f-bq5w2"]
Jul 01 17:20:51.618226 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:51.618203    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-1f194-predictor-6dd685749f-bq5w2" podUID="acb61295-6868-4bc3-9a75-77fae13527fd" containerName="kserve-container" containerID="cri-o://9e9afb6d6ed4a5b1219988d766ce10784798421fda448f6f7ac948eb8fed23ed" gracePeriod=30
Jul 01 17:20:51.618331 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:51.618299    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-1f194-predictor-6dd685749f-bq5w2" podUID="acb61295-6868-4bc3-9a75-77fae13527fd" containerName="kube-rbac-proxy" containerID="cri-o://72cd997375fd6d06c89d1e69bfdcc076823292d2817df9bb3ccbd10175b1fa91" gracePeriod=30
Jul 01 17:20:51.631025 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:51.627810    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6460c-predictor-5774675f9-x8h72"]
Jul 01 17:20:51.631025 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:51.628698    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e49336bc-9319-467b-a6b8-969be03815e0" containerName="kube-rbac-proxy"
Jul 01 17:20:51.631025 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:51.628718    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e49336bc-9319-467b-a6b8-969be03815e0" containerName="kube-rbac-proxy"
Jul 01 17:20:51.631025 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:51.628746    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e49336bc-9319-467b-a6b8-969be03815e0" containerName="kserve-container"
Jul 01 17:20:51.631025 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:51.628756    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e49336bc-9319-467b-a6b8-969be03815e0" containerName="kserve-container"
Jul 01 17:20:51.631025 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:51.628788    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="430b61ee-095e-470f-a487-7d790277b37d" containerName="kube-rbac-proxy"
Jul 01 17:20:51.631025 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:51.628797    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="430b61ee-095e-470f-a487-7d790277b37d" containerName="kube-rbac-proxy"
Jul 01 17:20:51.631025 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:51.628810    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="430b61ee-095e-470f-a487-7d790277b37d" containerName="kserve-container"
Jul 01 17:20:51.631025 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:51.628826    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="430b61ee-095e-470f-a487-7d790277b37d" containerName="kserve-container"
Jul 01 17:20:51.631025 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:51.628973    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="430b61ee-095e-470f-a487-7d790277b37d" containerName="kserve-container"
Jul 01 17:20:51.631025 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:51.628990    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="430b61ee-095e-470f-a487-7d790277b37d" containerName="kube-rbac-proxy"
Jul 01 17:20:51.631025 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:51.629001    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e49336bc-9319-467b-a6b8-969be03815e0" containerName="kube-rbac-proxy"
Jul 01 17:20:51.631025 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:51.629012    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e49336bc-9319-467b-a6b8-969be03815e0" containerName="kserve-container"
Jul 01 17:20:51.635339 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:51.635319    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-6460c-predictor-5774675f9-x8h72"
Jul 01 17:20:51.638143 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:51.638122    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-6460c-predictor-serving-cert\""
Jul 01 17:20:51.638253 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:51.638161    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-6460c-kube-rbac-proxy-sar-config\""
Jul 01 17:20:51.641399 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:51.641375    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6460c-predictor-5774675f9-x8h72"]
Jul 01 17:20:51.698764 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:51.698733    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:20:51.733846 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:51.733814    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5bb651ab-06a8-4dbc-812f-1eb6ac74f174-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-6460c-predictor-5774675f9-x8h72\" (UID: \"5bb651ab-06a8-4dbc-812f-1eb6ac74f174\") " pod="kserve-ci-e2e-test/success-200-isvc-6460c-predictor-5774675f9-x8h72"
Jul 01 17:20:51.733977 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:51.733878    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5bb651ab-06a8-4dbc-812f-1eb6ac74f174-proxy-tls\") pod \"success-200-isvc-6460c-predictor-5774675f9-x8h72\" (UID: \"5bb651ab-06a8-4dbc-812f-1eb6ac74f174\") " pod="kserve-ci-e2e-test/success-200-isvc-6460c-predictor-5774675f9-x8h72"
Jul 01 17:20:51.733977 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:51.733919    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8xc2\" (UniqueName: \"kubernetes.io/projected/5bb651ab-06a8-4dbc-812f-1eb6ac74f174-kube-api-access-q8xc2\") pod \"success-200-isvc-6460c-predictor-5774675f9-x8h72\" (UID: \"5bb651ab-06a8-4dbc-812f-1eb6ac74f174\") " pod="kserve-ci-e2e-test/success-200-isvc-6460c-predictor-5774675f9-x8h72"
Jul 01 17:20:51.738167 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:51.738104    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6460c-predictor-5d5d94dfb6-btq8d"]
Jul 01 17:20:51.741956 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:51.741939    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-6460c-predictor-5d5d94dfb6-btq8d"
Jul 01 17:20:51.747468 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:51.747413    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-6460c-kube-rbac-proxy-sar-config\""
Jul 01 17:20:51.747603 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:51.747561    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-6460c-predictor-serving-cert\""
Jul 01 17:20:51.755784 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:51.755758    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6460c-predictor-5d5d94dfb6-btq8d"]
Jul 01 17:20:51.778098 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:51.778067    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:20:51.835078 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:51.835040    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6b534f1-c10d-494c-8547-05f4a1073b40-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-6460c-predictor-5d5d94dfb6-btq8d\" (UID: \"c6b534f1-c10d-494c-8547-05f4a1073b40\") " pod="kserve-ci-e2e-test/error-404-isvc-6460c-predictor-5d5d94dfb6-btq8d"
Jul 01 17:20:51.835250 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:51.835121    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5bb651ab-06a8-4dbc-812f-1eb6ac74f174-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-6460c-predictor-5774675f9-x8h72\" (UID: \"5bb651ab-06a8-4dbc-812f-1eb6ac74f174\") " pod="kserve-ci-e2e-test/success-200-isvc-6460c-predictor-5774675f9-x8h72"
Jul 01 17:20:51.835250 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:51.835160    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5bb651ab-06a8-4dbc-812f-1eb6ac74f174-proxy-tls\") pod \"success-200-isvc-6460c-predictor-5774675f9-x8h72\" (UID: \"5bb651ab-06a8-4dbc-812f-1eb6ac74f174\") " pod="kserve-ci-e2e-test/success-200-isvc-6460c-predictor-5774675f9-x8h72"
Jul 01 17:20:51.835250 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:51.835179    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7jrn\" (UniqueName: \"kubernetes.io/projected/c6b534f1-c10d-494c-8547-05f4a1073b40-kube-api-access-w7jrn\") pod \"error-404-isvc-6460c-predictor-5d5d94dfb6-btq8d\" (UID: \"c6b534f1-c10d-494c-8547-05f4a1073b40\") " pod="kserve-ci-e2e-test/error-404-isvc-6460c-predictor-5d5d94dfb6-btq8d"
Jul 01 17:20:51.835250 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:51.835210    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8xc2\" (UniqueName: \"kubernetes.io/projected/5bb651ab-06a8-4dbc-812f-1eb6ac74f174-kube-api-access-q8xc2\") pod \"success-200-isvc-6460c-predictor-5774675f9-x8h72\" (UID: \"5bb651ab-06a8-4dbc-812f-1eb6ac74f174\") " pod="kserve-ci-e2e-test/success-200-isvc-6460c-predictor-5774675f9-x8h72"
Jul 01 17:20:51.835495 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:20:51.835323    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-6460c-predictor-serving-cert: secret "success-200-isvc-6460c-predictor-serving-cert" not found
Jul 01 17:20:51.835495 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:51.835393    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6b534f1-c10d-494c-8547-05f4a1073b40-proxy-tls\") pod \"error-404-isvc-6460c-predictor-5d5d94dfb6-btq8d\" (UID: \"c6b534f1-c10d-494c-8547-05f4a1073b40\") " pod="kserve-ci-e2e-test/error-404-isvc-6460c-predictor-5d5d94dfb6-btq8d"
Jul 01 17:20:51.835495 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:20:51.835422    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bb651ab-06a8-4dbc-812f-1eb6ac74f174-proxy-tls podName:5bb651ab-06a8-4dbc-812f-1eb6ac74f174 nodeName:}" failed. No retries permitted until 2026-07-01 17:20:52.335400587 +0000 UTC m=+1370.132538971 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5bb651ab-06a8-4dbc-812f-1eb6ac74f174-proxy-tls") pod "success-200-isvc-6460c-predictor-5774675f9-x8h72" (UID: "5bb651ab-06a8-4dbc-812f-1eb6ac74f174") : secret "success-200-isvc-6460c-predictor-serving-cert" not found
Jul 01 17:20:51.835898 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:51.835881    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5bb651ab-06a8-4dbc-812f-1eb6ac74f174-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-6460c-predictor-5774675f9-x8h72\" (UID: \"5bb651ab-06a8-4dbc-812f-1eb6ac74f174\") " pod="kserve-ci-e2e-test/success-200-isvc-6460c-predictor-5774675f9-x8h72"
Jul 01 17:20:51.844990 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:51.844965    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8xc2\" (UniqueName: \"kubernetes.io/projected/5bb651ab-06a8-4dbc-812f-1eb6ac74f174-kube-api-access-q8xc2\") pod \"success-200-isvc-6460c-predictor-5774675f9-x8h72\" (UID: \"5bb651ab-06a8-4dbc-812f-1eb6ac74f174\") " pod="kserve-ci-e2e-test/success-200-isvc-6460c-predictor-5774675f9-x8h72"
Jul 01 17:20:51.936682 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:51.936644    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6b534f1-c10d-494c-8547-05f4a1073b40-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-6460c-predictor-5d5d94dfb6-btq8d\" (UID: \"c6b534f1-c10d-494c-8547-05f4a1073b40\") " pod="kserve-ci-e2e-test/error-404-isvc-6460c-predictor-5d5d94dfb6-btq8d"
Jul 01 17:20:51.936874 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:51.936740    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7jrn\" (UniqueName: \"kubernetes.io/projected/c6b534f1-c10d-494c-8547-05f4a1073b40-kube-api-access-w7jrn\") pod \"error-404-isvc-6460c-predictor-5d5d94dfb6-btq8d\" (UID: \"c6b534f1-c10d-494c-8547-05f4a1073b40\") " pod="kserve-ci-e2e-test/error-404-isvc-6460c-predictor-5d5d94dfb6-btq8d"
Jul 01 17:20:51.936874 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:51.936796    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6b534f1-c10d-494c-8547-05f4a1073b40-proxy-tls\") pod \"error-404-isvc-6460c-predictor-5d5d94dfb6-btq8d\" (UID: \"c6b534f1-c10d-494c-8547-05f4a1073b40\") " pod="kserve-ci-e2e-test/error-404-isvc-6460c-predictor-5d5d94dfb6-btq8d"
Jul 01 17:20:51.937364 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:51.937319    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6b534f1-c10d-494c-8547-05f4a1073b40-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-6460c-predictor-5d5d94dfb6-btq8d\" (UID: \"c6b534f1-c10d-494c-8547-05f4a1073b40\") " pod="kserve-ci-e2e-test/error-404-isvc-6460c-predictor-5d5d94dfb6-btq8d"
Jul 01 17:20:51.939202 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:51.939179    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6b534f1-c10d-494c-8547-05f4a1073b40-proxy-tls\") pod \"error-404-isvc-6460c-predictor-5d5d94dfb6-btq8d\" (UID: \"c6b534f1-c10d-494c-8547-05f4a1073b40\") " pod="kserve-ci-e2e-test/error-404-isvc-6460c-predictor-5d5d94dfb6-btq8d"
Jul 01 17:20:51.944817 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:51.944794    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7jrn\" (UniqueName: \"kubernetes.io/projected/c6b534f1-c10d-494c-8547-05f4a1073b40-kube-api-access-w7jrn\") pod \"error-404-isvc-6460c-predictor-5d5d94dfb6-btq8d\" (UID: \"c6b534f1-c10d-494c-8547-05f4a1073b40\") " pod="kserve-ci-e2e-test/error-404-isvc-6460c-predictor-5d5d94dfb6-btq8d"
Jul 01 17:20:51.955530 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:51.955497    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:20:52.052962 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:52.052877    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-6460c-predictor-5d5d94dfb6-btq8d"
Jul 01 17:20:52.056603 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:52.056578    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:20:52.159752 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:52.159722    2563 generic.go:356] "Generic (PLEG): container finished" podID="dc2febf1-bc84-4180-8f3c-da46de283e1b" containerID="34c308d4f10809a65bf679816d0f1b7e41d54324e6c09ed016a9236ee3d58cce" exitCode=2
Jul 01 17:20:52.159899 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:52.159797    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1f194-predictor-6c5d4fc46b-kk5sz" event={"ID":"dc2febf1-bc84-4180-8f3c-da46de283e1b","Type":"ContainerDied","Data":"34c308d4f10809a65bf679816d0f1b7e41d54324e6c09ed016a9236ee3d58cce"}
Jul 01 17:20:52.161778 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:52.161751    2563 generic.go:356] "Generic (PLEG): container finished" podID="acb61295-6868-4bc3-9a75-77fae13527fd" containerID="72cd997375fd6d06c89d1e69bfdcc076823292d2817df9bb3ccbd10175b1fa91" exitCode=2
Jul 01 17:20:52.161880 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:52.161820    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1f194-predictor-6dd685749f-bq5w2" event={"ID":"acb61295-6868-4bc3-9a75-77fae13527fd","Type":"ContainerDied","Data":"72cd997375fd6d06c89d1e69bfdcc076823292d2817df9bb3ccbd10175b1fa91"}
Jul 01 17:20:52.189630 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:52.189520    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6460c-predictor-5d5d94dfb6-btq8d"]
Jul 01 17:20:52.192459 ip-10-0-137-180 kubenswrapper[2563]: W0701 17:20:52.192430    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6b534f1_c10d_494c_8547_05f4a1073b40.slice/crio-e1e77a18390cef67ce95efa0928755b91669d939e6902cb9aba95bc12522aa6a WatchSource:0}: Error finding container e1e77a18390cef67ce95efa0928755b91669d939e6902cb9aba95bc12522aa6a: Status 404 returned error can't find the container with id e1e77a18390cef67ce95efa0928755b91669d939e6902cb9aba95bc12522aa6a
Jul 01 17:20:52.341460 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:52.341371    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5bb651ab-06a8-4dbc-812f-1eb6ac74f174-proxy-tls\") pod \"success-200-isvc-6460c-predictor-5774675f9-x8h72\" (UID: \"5bb651ab-06a8-4dbc-812f-1eb6ac74f174\") " pod="kserve-ci-e2e-test/success-200-isvc-6460c-predictor-5774675f9-x8h72"
Jul 01 17:20:52.343736 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:52.343716    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5bb651ab-06a8-4dbc-812f-1eb6ac74f174-proxy-tls\") pod \"success-200-isvc-6460c-predictor-5774675f9-x8h72\" (UID: \"5bb651ab-06a8-4dbc-812f-1eb6ac74f174\") " pod="kserve-ci-e2e-test/success-200-isvc-6460c-predictor-5774675f9-x8h72"
Jul 01 17:20:52.547398 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:52.547365    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-6460c-predictor-5774675f9-x8h72"
Jul 01 17:20:52.684520 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:52.684336    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6460c-predictor-5774675f9-x8h72"]
Jul 01 17:20:52.687292 ip-10-0-137-180 kubenswrapper[2563]: W0701 17:20:52.687259    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bb651ab_06a8_4dbc_812f_1eb6ac74f174.slice/crio-84d391b5e7913b20e251734366832e6400a1a4f6b3798591266bbaf4f5e05682 WatchSource:0}: Error finding container 84d391b5e7913b20e251734366832e6400a1a4f6b3798591266bbaf4f5e05682: Status 404 returned error can't find the container with id 84d391b5e7913b20e251734366832e6400a1a4f6b3798591266bbaf4f5e05682
Jul 01 17:20:53.056085 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:53.056050    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bbcac-predictor-7874fb45b7-bmwwk" podUID="19df0b11-647e-4826-99c7-a3016979c15b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 01 17:20:53.167669 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:53.167632    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6460c-predictor-5d5d94dfb6-btq8d" event={"ID":"c6b534f1-c10d-494c-8547-05f4a1073b40","Type":"ContainerStarted","Data":"480dde0a78af44494f13050f968554628cc80278be8f15cec90da3852c52fe92"}
Jul 01 17:20:53.167842 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:53.167691    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6460c-predictor-5d5d94dfb6-btq8d" event={"ID":"c6b534f1-c10d-494c-8547-05f4a1073b40","Type":"ContainerStarted","Data":"42eff41b2c0479a41b1f9f28f136bf2093444e12ecc5932aec535790da72479d"}
Jul 01 17:20:53.167842 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:53.167730    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-6460c-predictor-5d5d94dfb6-btq8d"
Jul 01 17:20:53.167842 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:53.167761    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6460c-predictor-5d5d94dfb6-btq8d" event={"ID":"c6b534f1-c10d-494c-8547-05f4a1073b40","Type":"ContainerStarted","Data":"e1e77a18390cef67ce95efa0928755b91669d939e6902cb9aba95bc12522aa6a"}
Jul 01 17:20:53.169313 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:53.169292    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6460c-predictor-5774675f9-x8h72" event={"ID":"5bb651ab-06a8-4dbc-812f-1eb6ac74f174","Type":"ContainerStarted","Data":"39069cb5512cbd7334cbc4d70984d4493cc4eace9eff5526dae1f2087a92482f"}
Jul 01 17:20:53.169378 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:53.169357    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6460c-predictor-5774675f9-x8h72" event={"ID":"5bb651ab-06a8-4dbc-812f-1eb6ac74f174","Type":"ContainerStarted","Data":"153961105a76fbfc43c3457a4fa2fe9bc3c9e202fc2691149f798539103ac196"}
Jul 01 17:20:53.169421 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:53.169388    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6460c-predictor-5774675f9-x8h72" event={"ID":"5bb651ab-06a8-4dbc-812f-1eb6ac74f174","Type":"ContainerStarted","Data":"84d391b5e7913b20e251734366832e6400a1a4f6b3798591266bbaf4f5e05682"}
Jul 01 17:20:53.169421 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:53.169417    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-6460c-predictor-5774675f9-x8h72"
Jul 01 17:20:53.188887 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:53.188838    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-6460c-predictor-5d5d94dfb6-btq8d" podStartSLOduration=2.188822061 podStartE2EDuration="2.188822061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:20:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:20:53.187671157 +0000 UTC m=+1370.984809558" watchObservedRunningTime="2026-07-01 17:20:53.188822061 +0000 UTC m=+1370.985960462"
Jul 01 17:20:53.206207 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:53.206156    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-6460c-predictor-5774675f9-x8h72" podStartSLOduration=2.206140091 podStartE2EDuration="2.206140091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:20:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:20:53.205215375 +0000 UTC m=+1371.002353801" watchObservedRunningTime="2026-07-01 17:20:53.206140091 +0000 UTC m=+1371.003278492"
Jul 01 17:20:53.798822 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:53.798780    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1f194-predictor-6c5d4fc46b-kk5sz" podUID="dc2febf1-bc84-4180-8f3c-da46de283e1b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused"
Jul 01 17:20:53.799202 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:53.798784    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1f194-predictor-6dd685749f-bq5w2" podUID="acb61295-6868-4bc3-9a75-77fae13527fd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused"
Jul 01 17:20:54.173904 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:54.173805    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-6460c-predictor-5774675f9-x8h72"
Jul 01 17:20:54.173904 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:54.173872    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-6460c-predictor-5d5d94dfb6-btq8d"
Jul 01 17:20:54.174940 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:54.174908    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6460c-predictor-5d5d94dfb6-btq8d" podUID="c6b534f1-c10d-494c-8547-05f4a1073b40" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 01 17:20:54.174940 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:54.174923    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6460c-predictor-5774675f9-x8h72" podUID="5bb651ab-06a8-4dbc-812f-1eb6ac74f174" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 01 17:20:54.966060 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:54.966036    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-1f194-predictor-6dd685749f-bq5w2"
Jul 01 17:20:55.064402 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:55.064366    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r8w2w\" (UniqueName: \"kubernetes.io/projected/acb61295-6868-4bc3-9a75-77fae13527fd-kube-api-access-r8w2w\") pod \"acb61295-6868-4bc3-9a75-77fae13527fd\" (UID: \"acb61295-6868-4bc3-9a75-77fae13527fd\") "
Jul 01 17:20:55.064590 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:55.064484    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/acb61295-6868-4bc3-9a75-77fae13527fd-kube-rbac-proxy-sar-config\") pod \"acb61295-6868-4bc3-9a75-77fae13527fd\" (UID: \"acb61295-6868-4bc3-9a75-77fae13527fd\") "
Jul 01 17:20:55.064590 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:55.064524    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/acb61295-6868-4bc3-9a75-77fae13527fd-proxy-tls\") pod \"acb61295-6868-4bc3-9a75-77fae13527fd\" (UID: \"acb61295-6868-4bc3-9a75-77fae13527fd\") "
Jul 01 17:20:55.064904 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:55.064877    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acb61295-6868-4bc3-9a75-77fae13527fd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "acb61295-6868-4bc3-9a75-77fae13527fd" (UID: "acb61295-6868-4bc3-9a75-77fae13527fd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:20:55.066793 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:55.066739    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acb61295-6868-4bc3-9a75-77fae13527fd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "acb61295-6868-4bc3-9a75-77fae13527fd" (UID: "acb61295-6868-4bc3-9a75-77fae13527fd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 17:20:55.066991 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:55.066966    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acb61295-6868-4bc3-9a75-77fae13527fd-kube-api-access-r8w2w" (OuterVolumeSpecName: "kube-api-access-r8w2w") pod "acb61295-6868-4bc3-9a75-77fae13527fd" (UID: "acb61295-6868-4bc3-9a75-77fae13527fd"). InnerVolumeSpecName "kube-api-access-r8w2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 17:20:55.070071 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:55.070041    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bbcac-predictor-5ff4fd5cc4-wtk7h" podUID="9b4cccca-1a19-448a-b584-acf00f272af2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 01 17:20:55.165629 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:55.165594    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/acb61295-6868-4bc3-9a75-77fae13527fd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:20:55.165629 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:55.165628    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/acb61295-6868-4bc3-9a75-77fae13527fd-proxy-tls\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:20:55.165629 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:55.165637    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8w2w\" (UniqueName: \"kubernetes.io/projected/acb61295-6868-4bc3-9a75-77fae13527fd-kube-api-access-r8w2w\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:20:55.180154 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:55.180122    2563 generic.go:356] "Generic (PLEG): container finished" podID="acb61295-6868-4bc3-9a75-77fae13527fd" containerID="9e9afb6d6ed4a5b1219988d766ce10784798421fda448f6f7ac948eb8fed23ed" exitCode=0
Jul 01 17:20:55.180300 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:55.180167    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-1f194-predictor-6dd685749f-bq5w2"
Jul 01 17:20:55.180300 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:55.180202    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1f194-predictor-6dd685749f-bq5w2" event={"ID":"acb61295-6868-4bc3-9a75-77fae13527fd","Type":"ContainerDied","Data":"9e9afb6d6ed4a5b1219988d766ce10784798421fda448f6f7ac948eb8fed23ed"}
Jul 01 17:20:55.180300 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:55.180258    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1f194-predictor-6dd685749f-bq5w2" event={"ID":"acb61295-6868-4bc3-9a75-77fae13527fd","Type":"ContainerDied","Data":"95816f29708bab823c35b1bd27d46553c014bae521f6bb70a034e34d3518c2f2"}
Jul 01 17:20:55.180300 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:55.180285    2563 scope.go:117] "RemoveContainer" containerID="72cd997375fd6d06c89d1e69bfdcc076823292d2817df9bb3ccbd10175b1fa91"
Jul 01 17:20:55.180989 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:55.180926    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6460c-predictor-5d5d94dfb6-btq8d" podUID="c6b534f1-c10d-494c-8547-05f4a1073b40" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 01 17:20:55.181125 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:55.180974    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6460c-predictor-5774675f9-x8h72" podUID="5bb651ab-06a8-4dbc-812f-1eb6ac74f174" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 01 17:20:55.189287 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:55.189268    2563 scope.go:117] "RemoveContainer" containerID="9e9afb6d6ed4a5b1219988d766ce10784798421fda448f6f7ac948eb8fed23ed"
Jul 01 17:20:55.197279 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:55.197263    2563 scope.go:117] "RemoveContainer" containerID="72cd997375fd6d06c89d1e69bfdcc076823292d2817df9bb3ccbd10175b1fa91"
Jul 01 17:20:55.197559 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:20:55.197541    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72cd997375fd6d06c89d1e69bfdcc076823292d2817df9bb3ccbd10175b1fa91\": container with ID starting with 72cd997375fd6d06c89d1e69bfdcc076823292d2817df9bb3ccbd10175b1fa91 not found: ID does not exist" containerID="72cd997375fd6d06c89d1e69bfdcc076823292d2817df9bb3ccbd10175b1fa91"
Jul 01 17:20:55.197612 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:55.197568    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72cd997375fd6d06c89d1e69bfdcc076823292d2817df9bb3ccbd10175b1fa91"} err="failed to get container status \"72cd997375fd6d06c89d1e69bfdcc076823292d2817df9bb3ccbd10175b1fa91\": rpc error: code = NotFound desc = could not find container \"72cd997375fd6d06c89d1e69bfdcc076823292d2817df9bb3ccbd10175b1fa91\": container with ID starting with 72cd997375fd6d06c89d1e69bfdcc076823292d2817df9bb3ccbd10175b1fa91 not found: ID does not exist"
Jul 01 17:20:55.197612 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:55.197582    2563 scope.go:117] "RemoveContainer" containerID="9e9afb6d6ed4a5b1219988d766ce10784798421fda448f6f7ac948eb8fed23ed"
Jul 01 17:20:55.197807 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:20:55.197792    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e9afb6d6ed4a5b1219988d766ce10784798421fda448f6f7ac948eb8fed23ed\": container with ID starting with 9e9afb6d6ed4a5b1219988d766ce10784798421fda448f6f7ac948eb8fed23ed not found: ID does not exist" containerID="9e9afb6d6ed4a5b1219988d766ce10784798421fda448f6f7ac948eb8fed23ed"
Jul 01 17:20:55.197846 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:55.197812    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e9afb6d6ed4a5b1219988d766ce10784798421fda448f6f7ac948eb8fed23ed"} err="failed to get container status \"9e9afb6d6ed4a5b1219988d766ce10784798421fda448f6f7ac948eb8fed23ed\": rpc error: code = NotFound desc = could not find container \"9e9afb6d6ed4a5b1219988d766ce10784798421fda448f6f7ac948eb8fed23ed\": container with ID starting with 9e9afb6d6ed4a5b1219988d766ce10784798421fda448f6f7ac948eb8fed23ed not found: ID does not exist"
Jul 01 17:20:55.204543 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:55.204524    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1f194-predictor-6dd685749f-bq5w2"]
Jul 01 17:20:55.209194 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:55.209161    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1f194-predictor-6dd685749f-bq5w2"]
Jul 01 17:20:55.487950 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:55.487926    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1f194-predictor-6c5d4fc46b-kk5sz"
Jul 01 17:20:55.670282 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:55.670196    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k9jhc\" (UniqueName: \"kubernetes.io/projected/dc2febf1-bc84-4180-8f3c-da46de283e1b-kube-api-access-k9jhc\") pod \"dc2febf1-bc84-4180-8f3c-da46de283e1b\" (UID: \"dc2febf1-bc84-4180-8f3c-da46de283e1b\") "
Jul 01 17:20:55.670282 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:55.670232    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc2febf1-bc84-4180-8f3c-da46de283e1b-proxy-tls\") pod \"dc2febf1-bc84-4180-8f3c-da46de283e1b\" (UID: \"dc2febf1-bc84-4180-8f3c-da46de283e1b\") "
Jul 01 17:20:55.670282 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:55.670271    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc2febf1-bc84-4180-8f3c-da46de283e1b-kube-rbac-proxy-sar-config\") pod \"dc2febf1-bc84-4180-8f3c-da46de283e1b\" (UID: \"dc2febf1-bc84-4180-8f3c-da46de283e1b\") "
Jul 01 17:20:55.670700 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:55.670671    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc2febf1-bc84-4180-8f3c-da46de283e1b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "dc2febf1-bc84-4180-8f3c-da46de283e1b" (UID: "dc2febf1-bc84-4180-8f3c-da46de283e1b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:20:55.672388 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:55.672367    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc2febf1-bc84-4180-8f3c-da46de283e1b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dc2febf1-bc84-4180-8f3c-da46de283e1b" (UID: "dc2febf1-bc84-4180-8f3c-da46de283e1b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 17:20:55.672455 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:55.672424    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc2febf1-bc84-4180-8f3c-da46de283e1b-kube-api-access-k9jhc" (OuterVolumeSpecName: "kube-api-access-k9jhc") pod "dc2febf1-bc84-4180-8f3c-da46de283e1b" (UID: "dc2febf1-bc84-4180-8f3c-da46de283e1b"). InnerVolumeSpecName "kube-api-access-k9jhc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 17:20:55.771263 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:55.771218    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9jhc\" (UniqueName: \"kubernetes.io/projected/dc2febf1-bc84-4180-8f3c-da46de283e1b-kube-api-access-k9jhc\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:20:55.771263 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:55.771256    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc2febf1-bc84-4180-8f3c-da46de283e1b-proxy-tls\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:20:55.771263 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:55.771267    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc2febf1-bc84-4180-8f3c-da46de283e1b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:20:56.185946 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:56.185908    2563 generic.go:356] "Generic (PLEG): container finished" podID="dc2febf1-bc84-4180-8f3c-da46de283e1b" containerID="921fa68fcb3fcb73af90ff3ad0ee836e579e16f40df0df682dc03fc16d995ccd" exitCode=0
Jul 01 17:20:56.186440 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:56.185962    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1f194-predictor-6c5d4fc46b-kk5sz"
Jul 01 17:20:56.186440 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:56.186006    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1f194-predictor-6c5d4fc46b-kk5sz" event={"ID":"dc2febf1-bc84-4180-8f3c-da46de283e1b","Type":"ContainerDied","Data":"921fa68fcb3fcb73af90ff3ad0ee836e579e16f40df0df682dc03fc16d995ccd"}
Jul 01 17:20:56.186440 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:56.186083    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1f194-predictor-6c5d4fc46b-kk5sz" event={"ID":"dc2febf1-bc84-4180-8f3c-da46de283e1b","Type":"ContainerDied","Data":"394104227d0f9e5eaa5f79afeb1d4a30657fcf8b255af02454ce980608ca189f"}
Jul 01 17:20:56.186440 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:56.186128    2563 scope.go:117] "RemoveContainer" containerID="34c308d4f10809a65bf679816d0f1b7e41d54324e6c09ed016a9236ee3d58cce"
Jul 01 17:20:56.195116 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:56.195098    2563 scope.go:117] "RemoveContainer" containerID="921fa68fcb3fcb73af90ff3ad0ee836e579e16f40df0df682dc03fc16d995ccd"
Jul 01 17:20:56.204692 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:56.204671    2563 scope.go:117] "RemoveContainer" containerID="34c308d4f10809a65bf679816d0f1b7e41d54324e6c09ed016a9236ee3d58cce"
Jul 01 17:20:56.205169 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:20:56.205146    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34c308d4f10809a65bf679816d0f1b7e41d54324e6c09ed016a9236ee3d58cce\": container with ID starting with 34c308d4f10809a65bf679816d0f1b7e41d54324e6c09ed016a9236ee3d58cce not found: ID does not exist" containerID="34c308d4f10809a65bf679816d0f1b7e41d54324e6c09ed016a9236ee3d58cce"
Jul 01 17:20:56.205262 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:56.205179    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34c308d4f10809a65bf679816d0f1b7e41d54324e6c09ed016a9236ee3d58cce"} err="failed to get container status \"34c308d4f10809a65bf679816d0f1b7e41d54324e6c09ed016a9236ee3d58cce\": rpc error: code = NotFound desc = could not find container \"34c308d4f10809a65bf679816d0f1b7e41d54324e6c09ed016a9236ee3d58cce\": container with ID starting with 34c308d4f10809a65bf679816d0f1b7e41d54324e6c09ed016a9236ee3d58cce not found: ID does not exist"
Jul 01 17:20:56.205262 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:56.205201    2563 scope.go:117] "RemoveContainer" containerID="921fa68fcb3fcb73af90ff3ad0ee836e579e16f40df0df682dc03fc16d995ccd"
Jul 01 17:20:56.205513 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:20:56.205494    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"921fa68fcb3fcb73af90ff3ad0ee836e579e16f40df0df682dc03fc16d995ccd\": container with ID starting with 921fa68fcb3fcb73af90ff3ad0ee836e579e16f40df0df682dc03fc16d995ccd not found: ID does not exist" containerID="921fa68fcb3fcb73af90ff3ad0ee836e579e16f40df0df682dc03fc16d995ccd"
Jul 01 17:20:56.205587 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:56.205518    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"921fa68fcb3fcb73af90ff3ad0ee836e579e16f40df0df682dc03fc16d995ccd"} err="failed to get container status \"921fa68fcb3fcb73af90ff3ad0ee836e579e16f40df0df682dc03fc16d995ccd\": rpc error: code = NotFound desc = could not find container \"921fa68fcb3fcb73af90ff3ad0ee836e579e16f40df0df682dc03fc16d995ccd\": container with ID starting with 921fa68fcb3fcb73af90ff3ad0ee836e579e16f40df0df682dc03fc16d995ccd not found: ID does not exist"
Jul 01 17:20:56.211140 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:56.211116    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1f194-predictor-6c5d4fc46b-kk5sz"]
Jul 01 17:20:56.218920 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:56.218893    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1f194-predictor-6c5d4fc46b-kk5sz"]
Jul 01 17:20:56.697220 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:56.697182    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="acb61295-6868-4bc3-9a75-77fae13527fd" path="/var/lib/kubelet/pods/acb61295-6868-4bc3-9a75-77fae13527fd/volumes"
Jul 01 17:20:56.697682 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:20:56.697665    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc2febf1-bc84-4180-8f3c-da46de283e1b" path="/var/lib/kubelet/pods/dc2febf1-bc84-4180-8f3c-da46de283e1b/volumes"
Jul 01 17:21:00.186173 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:21:00.186139    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-6460c-predictor-5d5d94dfb6-btq8d"
Jul 01 17:21:00.186712 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:21:00.186226    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-6460c-predictor-5774675f9-x8h72"
Jul 01 17:21:00.186761 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:21:00.186719    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6460c-predictor-5774675f9-x8h72" podUID="5bb651ab-06a8-4dbc-812f-1eb6ac74f174" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 01 17:21:00.186761 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:21:00.186736    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6460c-predictor-5d5d94dfb6-btq8d" podUID="c6b534f1-c10d-494c-8547-05f4a1073b40" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 01 17:21:03.055864 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:21:03.055774    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bbcac-predictor-7874fb45b7-bmwwk" podUID="19df0b11-647e-4826-99c7-a3016979c15b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 01 17:21:05.069831 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:21:05.069792    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bbcac-predictor-5ff4fd5cc4-wtk7h" podUID="9b4cccca-1a19-448a-b584-acf00f272af2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 01 17:21:10.187046 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:21:10.187005    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6460c-predictor-5d5d94dfb6-btq8d" podUID="c6b534f1-c10d-494c-8547-05f4a1073b40" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 01 17:21:10.187479 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:21:10.187011    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6460c-predictor-5774675f9-x8h72" podUID="5bb651ab-06a8-4dbc-812f-1eb6ac74f174" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 01 17:21:13.056468 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:21:13.056436    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-bbcac-predictor-7874fb45b7-bmwwk"
Jul 01 17:21:15.071058 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:21:15.071029    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-bbcac-predictor-5ff4fd5cc4-wtk7h"
Jul 01 17:21:20.187241 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:21:20.187202    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6460c-predictor-5d5d94dfb6-btq8d" podUID="c6b534f1-c10d-494c-8547-05f4a1073b40" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 01 17:21:20.187669 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:21:20.187203    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6460c-predictor-5774675f9-x8h72" podUID="5bb651ab-06a8-4dbc-812f-1eb6ac74f174" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 01 17:21:25.902112 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:21:25.902053    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:21:26.507954 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:21:26.507916    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:21:30.187132 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:21:30.187097    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6460c-predictor-5d5d94dfb6-btq8d" podUID="c6b534f1-c10d-494c-8547-05f4a1073b40" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 01 17:21:30.187564 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:21:30.187094    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6460c-predictor-5774675f9-x8h72" podUID="5bb651ab-06a8-4dbc-812f-1eb6ac74f174" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 01 17:21:40.187332 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:21:40.187302    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-6460c-predictor-5774675f9-x8h72"
Jul 01 17:21:40.187838 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:21:40.187786    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-6460c-predictor-5d5d94dfb6-btq8d"
Jul 01 17:21:51.811205 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:21:51.811166    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:21:53.614287 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:21:53.614255    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:22:23.847924 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:22:23.847890    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:22:46.943066 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:22:46.943020    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:25:00.851476 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:25:00.851396    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:29:40.508557 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:40.508513    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:29:40.574131 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:40.574100    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bbcac-predictor-7874fb45b7-bmwwk"]
Jul 01 17:29:40.574397 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:40.574369    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-bbcac-predictor-7874fb45b7-bmwwk" podUID="19df0b11-647e-4826-99c7-a3016979c15b" containerName="kserve-container" containerID="cri-o://f9a9355f125fe56738b38a716843c3e98d4dba400d4e5ec94283f93bec92ea19" gracePeriod=30
Jul 01 17:29:40.574505 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:40.574380    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-bbcac-predictor-7874fb45b7-bmwwk" podUID="19df0b11-647e-4826-99c7-a3016979c15b" containerName="kube-rbac-proxy" containerID="cri-o://72652a651bf57110bb580b3cf4a116557e165c3296299f04d84b893f76d6ecd2" gracePeriod=30
Jul 01 17:29:40.640093 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:40.640056    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4517b-predictor-64b9b9f5bc-9pcrm"]
Jul 01 17:29:40.640426 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:40.640414    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc2febf1-bc84-4180-8f3c-da46de283e1b" containerName="kserve-container"
Jul 01 17:29:40.640482 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:40.640427    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc2febf1-bc84-4180-8f3c-da46de283e1b" containerName="kserve-container"
Jul 01 17:29:40.640482 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:40.640444    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acb61295-6868-4bc3-9a75-77fae13527fd" containerName="kserve-container"
Jul 01 17:29:40.640482 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:40.640449    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="acb61295-6868-4bc3-9a75-77fae13527fd" containerName="kserve-container"
Jul 01 17:29:40.640482 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:40.640455    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc2febf1-bc84-4180-8f3c-da46de283e1b" containerName="kube-rbac-proxy"
Jul 01 17:29:40.640482 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:40.640461    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc2febf1-bc84-4180-8f3c-da46de283e1b" containerName="kube-rbac-proxy"
Jul 01 17:29:40.640482 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:40.640467    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acb61295-6868-4bc3-9a75-77fae13527fd" containerName="kube-rbac-proxy"
Jul 01 17:29:40.640482 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:40.640473    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="acb61295-6868-4bc3-9a75-77fae13527fd" containerName="kube-rbac-proxy"
Jul 01 17:29:40.640689 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:40.640528    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="acb61295-6868-4bc3-9a75-77fae13527fd" containerName="kserve-container"
Jul 01 17:29:40.640689 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:40.640537    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc2febf1-bc84-4180-8f3c-da46de283e1b" containerName="kube-rbac-proxy"
Jul 01 17:29:40.640689 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:40.640546    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc2febf1-bc84-4180-8f3c-da46de283e1b" containerName="kserve-container"
Jul 01 17:29:40.640689 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:40.640552    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="acb61295-6868-4bc3-9a75-77fae13527fd" containerName="kube-rbac-proxy"
Jul 01 17:29:40.643564 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:40.643548    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4517b-predictor-64b9b9f5bc-9pcrm"
Jul 01 17:29:40.646278 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:40.646255    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-4517b-kube-rbac-proxy-sar-config\""
Jul 01 17:29:40.646528 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:40.646381    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-4517b-predictor-serving-cert\""
Jul 01 17:29:40.653936 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:40.653911    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4517b-predictor-64b9b9f5bc-9pcrm"]
Jul 01 17:29:40.703567 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:40.703540    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4517b-predictor-677799f794-kj8t9"]
Jul 01 17:29:40.706913 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:40.706897    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4517b-predictor-677799f794-kj8t9"
Jul 01 17:29:40.709503 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:40.709482    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-4517b-predictor-serving-cert\""
Jul 01 17:29:40.709752 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:40.709736    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-4517b-kube-rbac-proxy-sar-config\""
Jul 01 17:29:40.718052 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:40.718033    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4517b-predictor-677799f794-kj8t9"]
Jul 01 17:29:40.732374 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:40.732326    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/834629ee-e276-4d11-934b-2b6a7179ee66-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-4517b-predictor-64b9b9f5bc-9pcrm\" (UID: \"834629ee-e276-4d11-934b-2b6a7179ee66\") " pod="kserve-ci-e2e-test/success-200-isvc-4517b-predictor-64b9b9f5bc-9pcrm"
Jul 01 17:29:40.732606 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:40.732582    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/834629ee-e276-4d11-934b-2b6a7179ee66-proxy-tls\") pod \"success-200-isvc-4517b-predictor-64b9b9f5bc-9pcrm\" (UID: \"834629ee-e276-4d11-934b-2b6a7179ee66\") " pod="kserve-ci-e2e-test/success-200-isvc-4517b-predictor-64b9b9f5bc-9pcrm"
Jul 01 17:29:40.732813 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:40.732715    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44x89\" (UniqueName: \"kubernetes.io/projected/834629ee-e276-4d11-934b-2b6a7179ee66-kube-api-access-44x89\") pod \"success-200-isvc-4517b-predictor-64b9b9f5bc-9pcrm\" (UID: \"834629ee-e276-4d11-934b-2b6a7179ee66\") " pod="kserve-ci-e2e-test/success-200-isvc-4517b-predictor-64b9b9f5bc-9pcrm"
Jul 01 17:29:40.749619 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:40.749593    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bbcac-predictor-5ff4fd5cc4-wtk7h"]
Jul 01 17:29:40.749831 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:40.749805    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-bbcac-predictor-5ff4fd5cc4-wtk7h" podUID="9b4cccca-1a19-448a-b584-acf00f272af2" containerName="kserve-container" containerID="cri-o://d0aa041de1e5bdae243c0daeb941e569c5bd4c7e1a1ce982cacee42828cd25bf" gracePeriod=30
Jul 01 17:29:40.749906 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:40.749867    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-bbcac-predictor-5ff4fd5cc4-wtk7h" podUID="9b4cccca-1a19-448a-b584-acf00f272af2" containerName="kube-rbac-proxy" containerID="cri-o://f9177a7b1605443abf5df136ab7e4cdaf04b1d4e62021480e8761e7387b34717" gracePeriod=30
Jul 01 17:29:40.834113 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:40.834078    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73bda8d8-398f-46d6-a526-be667395fad7-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-4517b-predictor-677799f794-kj8t9\" (UID: \"73bda8d8-398f-46d6-a526-be667395fad7\") " pod="kserve-ci-e2e-test/error-404-isvc-4517b-predictor-677799f794-kj8t9"
Jul 01 17:29:40.834113 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:40.834127    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73bda8d8-398f-46d6-a526-be667395fad7-proxy-tls\") pod \"error-404-isvc-4517b-predictor-677799f794-kj8t9\" (UID: \"73bda8d8-398f-46d6-a526-be667395fad7\") " pod="kserve-ci-e2e-test/error-404-isvc-4517b-predictor-677799f794-kj8t9"
Jul 01 17:29:40.834317 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:40.834156    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44x89\" (UniqueName: \"kubernetes.io/projected/834629ee-e276-4d11-934b-2b6a7179ee66-kube-api-access-44x89\") pod \"success-200-isvc-4517b-predictor-64b9b9f5bc-9pcrm\" (UID: \"834629ee-e276-4d11-934b-2b6a7179ee66\") " pod="kserve-ci-e2e-test/success-200-isvc-4517b-predictor-64b9b9f5bc-9pcrm"
Jul 01 17:29:40.834317 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:40.834253    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktv2r\" (UniqueName: \"kubernetes.io/projected/73bda8d8-398f-46d6-a526-be667395fad7-kube-api-access-ktv2r\") pod \"error-404-isvc-4517b-predictor-677799f794-kj8t9\" (UID: \"73bda8d8-398f-46d6-a526-be667395fad7\") " pod="kserve-ci-e2e-test/error-404-isvc-4517b-predictor-677799f794-kj8t9"
Jul 01 17:29:40.834317 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:40.834301    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/834629ee-e276-4d11-934b-2b6a7179ee66-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-4517b-predictor-64b9b9f5bc-9pcrm\" (UID: \"834629ee-e276-4d11-934b-2b6a7179ee66\") " pod="kserve-ci-e2e-test/success-200-isvc-4517b-predictor-64b9b9f5bc-9pcrm"
Jul 01 17:29:40.834486 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:40.834374    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/834629ee-e276-4d11-934b-2b6a7179ee66-proxy-tls\") pod \"success-200-isvc-4517b-predictor-64b9b9f5bc-9pcrm\" (UID: \"834629ee-e276-4d11-934b-2b6a7179ee66\") " pod="kserve-ci-e2e-test/success-200-isvc-4517b-predictor-64b9b9f5bc-9pcrm"
Jul 01 17:29:40.834933 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:40.834914    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/834629ee-e276-4d11-934b-2b6a7179ee66-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-4517b-predictor-64b9b9f5bc-9pcrm\" (UID: \"834629ee-e276-4d11-934b-2b6a7179ee66\") " pod="kserve-ci-e2e-test/success-200-isvc-4517b-predictor-64b9b9f5bc-9pcrm"
Jul 01 17:29:40.836940 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:40.836917    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/834629ee-e276-4d11-934b-2b6a7179ee66-proxy-tls\") pod \"success-200-isvc-4517b-predictor-64b9b9f5bc-9pcrm\" (UID: \"834629ee-e276-4d11-934b-2b6a7179ee66\") " pod="kserve-ci-e2e-test/success-200-isvc-4517b-predictor-64b9b9f5bc-9pcrm"
Jul 01 17:29:40.842927 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:40.842908    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44x89\" (UniqueName: \"kubernetes.io/projected/834629ee-e276-4d11-934b-2b6a7179ee66-kube-api-access-44x89\") pod \"success-200-isvc-4517b-predictor-64b9b9f5bc-9pcrm\" (UID: \"834629ee-e276-4d11-934b-2b6a7179ee66\") " pod="kserve-ci-e2e-test/success-200-isvc-4517b-predictor-64b9b9f5bc-9pcrm"
Jul 01 17:29:40.935257 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:40.935223    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73bda8d8-398f-46d6-a526-be667395fad7-proxy-tls\") pod \"error-404-isvc-4517b-predictor-677799f794-kj8t9\" (UID: \"73bda8d8-398f-46d6-a526-be667395fad7\") " pod="kserve-ci-e2e-test/error-404-isvc-4517b-predictor-677799f794-kj8t9"
Jul 01 17:29:40.935429 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:40.935364    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktv2r\" (UniqueName: \"kubernetes.io/projected/73bda8d8-398f-46d6-a526-be667395fad7-kube-api-access-ktv2r\") pod \"error-404-isvc-4517b-predictor-677799f794-kj8t9\" (UID: \"73bda8d8-398f-46d6-a526-be667395fad7\") " pod="kserve-ci-e2e-test/error-404-isvc-4517b-predictor-677799f794-kj8t9"
Jul 01 17:29:40.935429 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:29:40.935416    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-4517b-predictor-serving-cert: secret "error-404-isvc-4517b-predictor-serving-cert" not found
Jul 01 17:29:40.935548 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:29:40.935492    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73bda8d8-398f-46d6-a526-be667395fad7-proxy-tls podName:73bda8d8-398f-46d6-a526-be667395fad7 nodeName:}" failed. No retries permitted until 2026-07-01 17:29:41.435470074 +0000 UTC m=+1899.232608453 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/73bda8d8-398f-46d6-a526-be667395fad7-proxy-tls") pod "error-404-isvc-4517b-predictor-677799f794-kj8t9" (UID: "73bda8d8-398f-46d6-a526-be667395fad7") : secret "error-404-isvc-4517b-predictor-serving-cert" not found
Jul 01 17:29:40.935548 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:40.935424    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73bda8d8-398f-46d6-a526-be667395fad7-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-4517b-predictor-677799f794-kj8t9\" (UID: \"73bda8d8-398f-46d6-a526-be667395fad7\") " pod="kserve-ci-e2e-test/error-404-isvc-4517b-predictor-677799f794-kj8t9"
Jul 01 17:29:40.936173 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:40.936151    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73bda8d8-398f-46d6-a526-be667395fad7-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-4517b-predictor-677799f794-kj8t9\" (UID: \"73bda8d8-398f-46d6-a526-be667395fad7\") " pod="kserve-ci-e2e-test/error-404-isvc-4517b-predictor-677799f794-kj8t9"
Jul 01 17:29:40.946626 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:40.946606    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktv2r\" (UniqueName: \"kubernetes.io/projected/73bda8d8-398f-46d6-a526-be667395fad7-kube-api-access-ktv2r\") pod \"error-404-isvc-4517b-predictor-677799f794-kj8t9\" (UID: \"73bda8d8-398f-46d6-a526-be667395fad7\") " pod="kserve-ci-e2e-test/error-404-isvc-4517b-predictor-677799f794-kj8t9"
Jul 01 17:29:40.957500 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:40.957479    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4517b-predictor-64b9b9f5bc-9pcrm"
Jul 01 17:29:41.098104 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:41.098051    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4517b-predictor-64b9b9f5bc-9pcrm"]
Jul 01 17:29:41.100695 ip-10-0-137-180 kubenswrapper[2563]: W0701 17:29:41.100669    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod834629ee_e276_4d11_934b_2b6a7179ee66.slice/crio-d5145e3076658ebe7a51cc878e5fdead81770db4956b2ed5a8615d62e8130a5d WatchSource:0}: Error finding container d5145e3076658ebe7a51cc878e5fdead81770db4956b2ed5a8615d62e8130a5d: Status 404 returned error can't find the container with id d5145e3076658ebe7a51cc878e5fdead81770db4956b2ed5a8615d62e8130a5d
Jul 01 17:29:41.283414 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:41.283379    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4517b-predictor-64b9b9f5bc-9pcrm" event={"ID":"834629ee-e276-4d11-934b-2b6a7179ee66","Type":"ContainerStarted","Data":"938988ab4bedc0e898ad8ecac1c8f05472a3e6ed1344958b8626959144c3db6e"}
Jul 01 17:29:41.283608 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:41.283448    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-4517b-predictor-64b9b9f5bc-9pcrm"
Jul 01 17:29:41.283608 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:41.283478    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4517b-predictor-64b9b9f5bc-9pcrm" event={"ID":"834629ee-e276-4d11-934b-2b6a7179ee66","Type":"ContainerStarted","Data":"2a22ccc9d7a74de3ed9f6307c617fb5ad4c9a58d9d5173bf45e3ca8635b03002"}
Jul 01 17:29:41.283726 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:41.283631    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-4517b-predictor-64b9b9f5bc-9pcrm"
Jul 01 17:29:41.283726 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:41.283688    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4517b-predictor-64b9b9f5bc-9pcrm" event={"ID":"834629ee-e276-4d11-934b-2b6a7179ee66","Type":"ContainerStarted","Data":"d5145e3076658ebe7a51cc878e5fdead81770db4956b2ed5a8615d62e8130a5d"}
Jul 01 17:29:41.285175 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:41.285150    2563 generic.go:356] "Generic (PLEG): container finished" podID="19df0b11-647e-4826-99c7-a3016979c15b" containerID="72652a651bf57110bb580b3cf4a116557e165c3296299f04d84b893f76d6ecd2" exitCode=2
Jul 01 17:29:41.285328 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:41.285215    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bbcac-predictor-7874fb45b7-bmwwk" event={"ID":"19df0b11-647e-4826-99c7-a3016979c15b","Type":"ContainerDied","Data":"72652a651bf57110bb580b3cf4a116557e165c3296299f04d84b893f76d6ecd2"}
Jul 01 17:29:41.285328 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:41.285272    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4517b-predictor-64b9b9f5bc-9pcrm" podUID="834629ee-e276-4d11-934b-2b6a7179ee66" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 01 17:29:41.286982 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:41.286960    2563 generic.go:356] "Generic (PLEG): container finished" podID="9b4cccca-1a19-448a-b584-acf00f272af2" containerID="f9177a7b1605443abf5df136ab7e4cdaf04b1d4e62021480e8761e7387b34717" exitCode=2
Jul 01 17:29:41.287068 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:41.287003    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bbcac-predictor-5ff4fd5cc4-wtk7h" event={"ID":"9b4cccca-1a19-448a-b584-acf00f272af2","Type":"ContainerDied","Data":"f9177a7b1605443abf5df136ab7e4cdaf04b1d4e62021480e8761e7387b34717"}
Jul 01 17:29:41.304895 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:41.304809    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-4517b-predictor-64b9b9f5bc-9pcrm" podStartSLOduration=1.3047930540000001 podStartE2EDuration="1.304793054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:29:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:29:41.302638448 +0000 UTC m=+1899.099776848" watchObservedRunningTime="2026-07-01 17:29:41.304793054 +0000 UTC m=+1899.101931455"
Jul 01 17:29:41.439647 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:41.439610    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73bda8d8-398f-46d6-a526-be667395fad7-proxy-tls\") pod \"error-404-isvc-4517b-predictor-677799f794-kj8t9\" (UID: \"73bda8d8-398f-46d6-a526-be667395fad7\") " pod="kserve-ci-e2e-test/error-404-isvc-4517b-predictor-677799f794-kj8t9"
Jul 01 17:29:41.442042 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:41.442022    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73bda8d8-398f-46d6-a526-be667395fad7-proxy-tls\") pod \"error-404-isvc-4517b-predictor-677799f794-kj8t9\" (UID: \"73bda8d8-398f-46d6-a526-be667395fad7\") " pod="kserve-ci-e2e-test/error-404-isvc-4517b-predictor-677799f794-kj8t9"
Jul 01 17:29:41.617648 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:41.617553    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4517b-predictor-677799f794-kj8t9"
Jul 01 17:29:41.751387 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:41.751362    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4517b-predictor-677799f794-kj8t9"]
Jul 01 17:29:41.753552 ip-10-0-137-180 kubenswrapper[2563]: W0701 17:29:41.753521    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73bda8d8_398f_46d6_a526_be667395fad7.slice/crio-137df8cf9fc1a952d49e05e37e0b4da843a464424c4d9e3e75a20354193fc062 WatchSource:0}: Error finding container 137df8cf9fc1a952d49e05e37e0b4da843a464424c4d9e3e75a20354193fc062: Status 404 returned error can't find the container with id 137df8cf9fc1a952d49e05e37e0b4da843a464424c4d9e3e75a20354193fc062
Jul 01 17:29:42.292461 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:42.292421    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4517b-predictor-677799f794-kj8t9" event={"ID":"73bda8d8-398f-46d6-a526-be667395fad7","Type":"ContainerStarted","Data":"60de77b307e4f9ee24a4428d99b45969256646b7030b7788b13446682d8ed8d5"}
Jul 01 17:29:42.292653 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:42.292495    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4517b-predictor-677799f794-kj8t9" event={"ID":"73bda8d8-398f-46d6-a526-be667395fad7","Type":"ContainerStarted","Data":"b88c8d76f8e3a0674ad8135a026550dd7b1e5ba8ccdd40202d2db514407fd16e"}
Jul 01 17:29:42.292653 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:42.292527    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4517b-predictor-677799f794-kj8t9" event={"ID":"73bda8d8-398f-46d6-a526-be667395fad7","Type":"ContainerStarted","Data":"137df8cf9fc1a952d49e05e37e0b4da843a464424c4d9e3e75a20354193fc062"}
Jul 01 17:29:42.292653 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:42.292634    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-4517b-predictor-677799f794-kj8t9"
Jul 01 17:29:42.292941 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:42.292915    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4517b-predictor-64b9b9f5bc-9pcrm" podUID="834629ee-e276-4d11-934b-2b6a7179ee66" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 01 17:29:42.315838 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:42.315790    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-4517b-predictor-677799f794-kj8t9" podStartSLOduration=2.315775561 podStartE2EDuration="2.315775561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:29:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:29:42.313701352 +0000 UTC m=+1900.110839772" watchObservedRunningTime="2026-07-01 17:29:42.315775561 +0000 UTC m=+1900.112913961"
Jul 01 17:29:42.935851 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:42.935815    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:29:43.050694 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:43.050653    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bbcac-predictor-7874fb45b7-bmwwk" podUID="19df0b11-647e-4826-99c7-a3016979c15b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused"
Jul 01 17:29:43.055689 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:43.055660    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bbcac-predictor-7874fb45b7-bmwwk" podUID="19df0b11-647e-4826-99c7-a3016979c15b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 01 17:29:43.296416 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:43.296384    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-4517b-predictor-677799f794-kj8t9"
Jul 01 17:29:43.297767 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:43.297737    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4517b-predictor-677799f794-kj8t9" podUID="73bda8d8-398f-46d6-a526-be667395fad7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 01 17:29:44.301743 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:44.301714    2563 generic.go:356] "Generic (PLEG): container finished" podID="19df0b11-647e-4826-99c7-a3016979c15b" containerID="f9a9355f125fe56738b38a716843c3e98d4dba400d4e5ec94283f93bec92ea19" exitCode=0
Jul 01 17:29:44.302153 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:44.301782    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bbcac-predictor-7874fb45b7-bmwwk" event={"ID":"19df0b11-647e-4826-99c7-a3016979c15b","Type":"ContainerDied","Data":"f9a9355f125fe56738b38a716843c3e98d4dba400d4e5ec94283f93bec92ea19"}
Jul 01 17:29:44.302338 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:44.302314    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4517b-predictor-677799f794-kj8t9" podUID="73bda8d8-398f-46d6-a526-be667395fad7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 01 17:29:44.437981 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:44.437958    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-bbcac-predictor-7874fb45b7-bmwwk"
Jul 01 17:29:44.506678 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:44.506653    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-bbcac-predictor-5ff4fd5cc4-wtk7h"
Jul 01 17:29:44.567535 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:44.567501    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ztwrx\" (UniqueName: \"kubernetes.io/projected/19df0b11-647e-4826-99c7-a3016979c15b-kube-api-access-ztwrx\") pod \"19df0b11-647e-4826-99c7-a3016979c15b\" (UID: \"19df0b11-647e-4826-99c7-a3016979c15b\") "
Jul 01 17:29:44.567535 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:44.567541    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19df0b11-647e-4826-99c7-a3016979c15b-proxy-tls\") pod \"19df0b11-647e-4826-99c7-a3016979c15b\" (UID: \"19df0b11-647e-4826-99c7-a3016979c15b\") "
Jul 01 17:29:44.567786 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:44.567577    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19df0b11-647e-4826-99c7-a3016979c15b-kube-rbac-proxy-sar-config\") pod \"19df0b11-647e-4826-99c7-a3016979c15b\" (UID: \"19df0b11-647e-4826-99c7-a3016979c15b\") "
Jul 01 17:29:44.567786 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:44.567706    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b4cccca-1a19-448a-b584-acf00f272af2-proxy-tls\") pod \"9b4cccca-1a19-448a-b584-acf00f272af2\" (UID: \"9b4cccca-1a19-448a-b584-acf00f272af2\") "
Jul 01 17:29:44.567786 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:44.567749    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9m6mt\" (UniqueName: \"kubernetes.io/projected/9b4cccca-1a19-448a-b584-acf00f272af2-kube-api-access-9m6mt\") pod \"9b4cccca-1a19-448a-b584-acf00f272af2\" (UID: \"9b4cccca-1a19-448a-b584-acf00f272af2\") "
Jul 01 17:29:44.567983 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:44.567799    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b4cccca-1a19-448a-b584-acf00f272af2-kube-rbac-proxy-sar-config\") pod \"9b4cccca-1a19-448a-b584-acf00f272af2\" (UID: \"9b4cccca-1a19-448a-b584-acf00f272af2\") "
Jul 01 17:29:44.567983 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:44.567888    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19df0b11-647e-4826-99c7-a3016979c15b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "19df0b11-647e-4826-99c7-a3016979c15b" (UID: "19df0b11-647e-4826-99c7-a3016979c15b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:29:44.568075 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:44.568059    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19df0b11-647e-4826-99c7-a3016979c15b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:29:44.568441 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:44.568246    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b4cccca-1a19-448a-b584-acf00f272af2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9b4cccca-1a19-448a-b584-acf00f272af2" (UID: "9b4cccca-1a19-448a-b584-acf00f272af2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:29:44.569877 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:44.569847    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19df0b11-647e-4826-99c7-a3016979c15b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "19df0b11-647e-4826-99c7-a3016979c15b" (UID: "19df0b11-647e-4826-99c7-a3016979c15b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 17:29:44.569977 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:44.569887    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b4cccca-1a19-448a-b584-acf00f272af2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9b4cccca-1a19-448a-b584-acf00f272af2" (UID: "9b4cccca-1a19-448a-b584-acf00f272af2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 17:29:44.569977 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:44.569910    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b4cccca-1a19-448a-b584-acf00f272af2-kube-api-access-9m6mt" (OuterVolumeSpecName: "kube-api-access-9m6mt") pod "9b4cccca-1a19-448a-b584-acf00f272af2" (UID: "9b4cccca-1a19-448a-b584-acf00f272af2"). InnerVolumeSpecName "kube-api-access-9m6mt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 17:29:44.569977 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:44.569916    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19df0b11-647e-4826-99c7-a3016979c15b-kube-api-access-ztwrx" (OuterVolumeSpecName: "kube-api-access-ztwrx") pod "19df0b11-647e-4826-99c7-a3016979c15b" (UID: "19df0b11-647e-4826-99c7-a3016979c15b"). InnerVolumeSpecName "kube-api-access-ztwrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 17:29:44.669244 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:44.669205    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b4cccca-1a19-448a-b584-acf00f272af2-proxy-tls\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:29:44.669244 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:44.669238    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9m6mt\" (UniqueName: \"kubernetes.io/projected/9b4cccca-1a19-448a-b584-acf00f272af2-kube-api-access-9m6mt\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:29:44.669244 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:44.669250    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b4cccca-1a19-448a-b584-acf00f272af2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:29:44.669475 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:44.669259    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztwrx\" (UniqueName: \"kubernetes.io/projected/19df0b11-647e-4826-99c7-a3016979c15b-kube-api-access-ztwrx\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:29:44.669475 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:44.669268    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19df0b11-647e-4826-99c7-a3016979c15b-proxy-tls\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:29:45.308114 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:45.308087    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-bbcac-predictor-7874fb45b7-bmwwk"
Jul 01 17:29:45.308571 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:45.308108    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bbcac-predictor-7874fb45b7-bmwwk" event={"ID":"19df0b11-647e-4826-99c7-a3016979c15b","Type":"ContainerDied","Data":"116c82a114b623aade2c6af9195d7295680741e88db055f7ca92993724647137"}
Jul 01 17:29:45.308571 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:45.308177    2563 scope.go:117] "RemoveContainer" containerID="72652a651bf57110bb580b3cf4a116557e165c3296299f04d84b893f76d6ecd2"
Jul 01 17:29:45.310331 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:45.310191    2563 generic.go:356] "Generic (PLEG): container finished" podID="9b4cccca-1a19-448a-b584-acf00f272af2" containerID="d0aa041de1e5bdae243c0daeb941e569c5bd4c7e1a1ce982cacee42828cd25bf" exitCode=0
Jul 01 17:29:45.310331 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:45.310234    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bbcac-predictor-5ff4fd5cc4-wtk7h" event={"ID":"9b4cccca-1a19-448a-b584-acf00f272af2","Type":"ContainerDied","Data":"d0aa041de1e5bdae243c0daeb941e569c5bd4c7e1a1ce982cacee42828cd25bf"}
Jul 01 17:29:45.310331 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:45.310286    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bbcac-predictor-5ff4fd5cc4-wtk7h" event={"ID":"9b4cccca-1a19-448a-b584-acf00f272af2","Type":"ContainerDied","Data":"c70deaf1d76e862a2ec16e0ca31a602442cccfac4bcfe87c9053ed9845e1fea3"}
Jul 01 17:29:45.310581 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:45.310339    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-bbcac-predictor-5ff4fd5cc4-wtk7h"
Jul 01 17:29:45.317320 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:45.317303    2563 scope.go:117] "RemoveContainer" containerID="f9a9355f125fe56738b38a716843c3e98d4dba400d4e5ec94283f93bec92ea19"
Jul 01 17:29:45.327489 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:45.327464    2563 scope.go:117] "RemoveContainer" containerID="f9177a7b1605443abf5df136ab7e4cdaf04b1d4e62021480e8761e7387b34717"
Jul 01 17:29:45.331760 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:45.331739    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bbcac-predictor-7874fb45b7-bmwwk"]
Jul 01 17:29:45.335301 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:45.335286    2563 scope.go:117] "RemoveContainer" containerID="d0aa041de1e5bdae243c0daeb941e569c5bd4c7e1a1ce982cacee42828cd25bf"
Jul 01 17:29:45.336506 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:45.336490    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bbcac-predictor-7874fb45b7-bmwwk"]
Jul 01 17:29:45.342687 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:45.342666    2563 scope.go:117] "RemoveContainer" containerID="f9177a7b1605443abf5df136ab7e4cdaf04b1d4e62021480e8761e7387b34717"
Jul 01 17:29:45.342952 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:29:45.342930    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9177a7b1605443abf5df136ab7e4cdaf04b1d4e62021480e8761e7387b34717\": container with ID starting with f9177a7b1605443abf5df136ab7e4cdaf04b1d4e62021480e8761e7387b34717 not found: ID does not exist" containerID="f9177a7b1605443abf5df136ab7e4cdaf04b1d4e62021480e8761e7387b34717"
Jul 01 17:29:45.343003 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:45.342962    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9177a7b1605443abf5df136ab7e4cdaf04b1d4e62021480e8761e7387b34717"} err="failed to get container status \"f9177a7b1605443abf5df136ab7e4cdaf04b1d4e62021480e8761e7387b34717\": rpc error: code = NotFound desc = could not find container \"f9177a7b1605443abf5df136ab7e4cdaf04b1d4e62021480e8761e7387b34717\": container with ID starting with f9177a7b1605443abf5df136ab7e4cdaf04b1d4e62021480e8761e7387b34717 not found: ID does not exist"
Jul 01 17:29:45.343003 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:45.342979    2563 scope.go:117] "RemoveContainer" containerID="d0aa041de1e5bdae243c0daeb941e569c5bd4c7e1a1ce982cacee42828cd25bf"
Jul 01 17:29:45.343182 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:29:45.343162    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0aa041de1e5bdae243c0daeb941e569c5bd4c7e1a1ce982cacee42828cd25bf\": container with ID starting with d0aa041de1e5bdae243c0daeb941e569c5bd4c7e1a1ce982cacee42828cd25bf not found: ID does not exist" containerID="d0aa041de1e5bdae243c0daeb941e569c5bd4c7e1a1ce982cacee42828cd25bf"
Jul 01 17:29:45.343226 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:45.343188    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0aa041de1e5bdae243c0daeb941e569c5bd4c7e1a1ce982cacee42828cd25bf"} err="failed to get container status \"d0aa041de1e5bdae243c0daeb941e569c5bd4c7e1a1ce982cacee42828cd25bf\": rpc error: code = NotFound desc = could not find container \"d0aa041de1e5bdae243c0daeb941e569c5bd4c7e1a1ce982cacee42828cd25bf\": container with ID starting with d0aa041de1e5bdae243c0daeb941e569c5bd4c7e1a1ce982cacee42828cd25bf not found: ID does not exist"
Jul 01 17:29:45.349784 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:45.349764    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bbcac-predictor-5ff4fd5cc4-wtk7h"]
Jul 01 17:29:45.355590 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:45.355570    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bbcac-predictor-5ff4fd5cc4-wtk7h"]
Jul 01 17:29:46.696231 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:46.696196    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19df0b11-647e-4826-99c7-a3016979c15b" path="/var/lib/kubelet/pods/19df0b11-647e-4826-99c7-a3016979c15b/volumes"
Jul 01 17:29:46.696627 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:46.696612    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b4cccca-1a19-448a-b584-acf00f272af2" path="/var/lib/kubelet/pods/9b4cccca-1a19-448a-b584-acf00f272af2/volumes"
Jul 01 17:29:47.297522 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:47.297496    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-4517b-predictor-64b9b9f5bc-9pcrm"
Jul 01 17:29:47.297889 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:47.297851    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4517b-predictor-64b9b9f5bc-9pcrm" podUID="834629ee-e276-4d11-934b-2b6a7179ee66" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 01 17:29:49.307208 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:49.307178    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-4517b-predictor-677799f794-kj8t9"
Jul 01 17:29:49.307739 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:49.307713    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4517b-predictor-677799f794-kj8t9" podUID="73bda8d8-398f-46d6-a526-be667395fad7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 01 17:29:57.298070 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:57.298027    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4517b-predictor-64b9b9f5bc-9pcrm" podUID="834629ee-e276-4d11-934b-2b6a7179ee66" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 01 17:29:59.307715 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:29:59.307674    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4517b-predictor-677799f794-kj8t9" podUID="73bda8d8-398f-46d6-a526-be667395fad7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 01 17:30:06.282051 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:06.282014    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:30:06.358033 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:06.357998    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6460c-predictor-5d5d94dfb6-btq8d"]
Jul 01 17:30:06.358297 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:06.358272    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-6460c-predictor-5d5d94dfb6-btq8d" podUID="c6b534f1-c10d-494c-8547-05f4a1073b40" containerName="kserve-container" containerID="cri-o://42eff41b2c0479a41b1f9f28f136bf2093444e12ecc5932aec535790da72479d" gracePeriod=30
Jul 01 17:30:06.358399 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:06.358329    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-6460c-predictor-5d5d94dfb6-btq8d" podUID="c6b534f1-c10d-494c-8547-05f4a1073b40" containerName="kube-rbac-proxy" containerID="cri-o://480dde0a78af44494f13050f968554628cc80278be8f15cec90da3852c52fe92" gracePeriod=30
Jul 01 17:30:06.411275 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:06.411238    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:30:06.444520 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:06.444482    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6460c-predictor-5774675f9-x8h72"]
Jul 01 17:30:06.444742 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:06.444692    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-6460c-predictor-5774675f9-x8h72" podUID="5bb651ab-06a8-4dbc-812f-1eb6ac74f174" containerName="kserve-container" containerID="cri-o://153961105a76fbfc43c3457a4fa2fe9bc3c9e202fc2691149f798539103ac196" gracePeriod=30
Jul 01 17:30:06.444822 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:06.444733    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-6460c-predictor-5774675f9-x8h72" podUID="5bb651ab-06a8-4dbc-812f-1eb6ac74f174" containerName="kube-rbac-proxy" containerID="cri-o://39069cb5512cbd7334cbc4d70984d4493cc4eace9eff5526dae1f2087a92482f" gracePeriod=30
Jul 01 17:30:06.504568 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:06.504536    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-db836-predictor-598457c8f9-bc6fv"]
Jul 01 17:30:06.504947 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:06.504930    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b4cccca-1a19-448a-b584-acf00f272af2" containerName="kube-rbac-proxy"
Jul 01 17:30:06.505034 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:06.504950    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b4cccca-1a19-448a-b584-acf00f272af2" containerName="kube-rbac-proxy"
Jul 01 17:30:06.505034 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:06.504962    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19df0b11-647e-4826-99c7-a3016979c15b" containerName="kserve-container"
Jul 01 17:30:06.505034 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:06.504971    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="19df0b11-647e-4826-99c7-a3016979c15b" containerName="kserve-container"
Jul 01 17:30:06.505034 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:06.504982    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b4cccca-1a19-448a-b584-acf00f272af2" containerName="kserve-container"
Jul 01 17:30:06.505034 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:06.504990    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b4cccca-1a19-448a-b584-acf00f272af2" containerName="kserve-container"
Jul 01 17:30:06.505034 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:06.505005    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19df0b11-647e-4826-99c7-a3016979c15b" containerName="kube-rbac-proxy"
Jul 01 17:30:06.505034 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:06.505013    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="19df0b11-647e-4826-99c7-a3016979c15b" containerName="kube-rbac-proxy"
Jul 01 17:30:06.505410 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:06.505109    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b4cccca-1a19-448a-b584-acf00f272af2" containerName="kube-rbac-proxy"
Jul 01 17:30:06.505410 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:06.505121    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="19df0b11-647e-4826-99c7-a3016979c15b" containerName="kserve-container"
Jul 01 17:30:06.505410 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:06.505132    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="19df0b11-647e-4826-99c7-a3016979c15b" containerName="kube-rbac-proxy"
Jul 01 17:30:06.505410 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:06.505141    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b4cccca-1a19-448a-b584-acf00f272af2" containerName="kserve-container"
Jul 01 17:30:06.508244 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:06.508225    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-db836-predictor-598457c8f9-bc6fv"
Jul 01 17:30:06.510981 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:06.510953    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-db836-predictor-serving-cert\""
Jul 01 17:30:06.511395 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:06.511378    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-db836-kube-rbac-proxy-sar-config\""
Jul 01 17:30:06.525047 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:06.525017    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-db836-predictor-598457c8f9-bc6fv"]
Jul 01 17:30:06.534367 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:06.534280    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:30:06.562727 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:06.562696    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s79fz\" (UniqueName: \"kubernetes.io/projected/0b838ab0-36ca-443c-a390-0323761a03fc-kube-api-access-s79fz\") pod \"success-200-isvc-db836-predictor-598457c8f9-bc6fv\" (UID: \"0b838ab0-36ca-443c-a390-0323761a03fc\") " pod="kserve-ci-e2e-test/success-200-isvc-db836-predictor-598457c8f9-bc6fv"
Jul 01 17:30:06.562876 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:06.562791    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b838ab0-36ca-443c-a390-0323761a03fc-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-db836-predictor-598457c8f9-bc6fv\" (UID: \"0b838ab0-36ca-443c-a390-0323761a03fc\") " pod="kserve-ci-e2e-test/success-200-isvc-db836-predictor-598457c8f9-bc6fv"
Jul 01 17:30:06.562876 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:06.562856    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b838ab0-36ca-443c-a390-0323761a03fc-proxy-tls\") pod \"success-200-isvc-db836-predictor-598457c8f9-bc6fv\" (UID: \"0b838ab0-36ca-443c-a390-0323761a03fc\") " pod="kserve-ci-e2e-test/success-200-isvc-db836-predictor-598457c8f9-bc6fv"
Jul 01 17:30:06.604988 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:06.604954    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-db836-predictor-6996c59df8-gc6q8"]
Jul 01 17:30:06.608315 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:06.608299    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-db836-predictor-6996c59df8-gc6q8"
Jul 01 17:30:06.611806 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:06.611786    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-db836-predictor-serving-cert\""
Jul 01 17:30:06.612071 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:06.612045    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-db836-kube-rbac-proxy-sar-config\""
Jul 01 17:30:06.619311 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:06.619284    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-db836-predictor-6996c59df8-gc6q8"]
Jul 01 17:30:06.663169 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:06.663133    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghvqs\" (UniqueName: \"kubernetes.io/projected/9a6ed0aa-189f-4c19-bfee-d052a47aa0e9-kube-api-access-ghvqs\") pod \"error-404-isvc-db836-predictor-6996c59df8-gc6q8\" (UID: \"9a6ed0aa-189f-4c19-bfee-d052a47aa0e9\") " pod="kserve-ci-e2e-test/error-404-isvc-db836-predictor-6996c59df8-gc6q8"
Jul 01 17:30:06.663378 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:06.663176    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s79fz\" (UniqueName: \"kubernetes.io/projected/0b838ab0-36ca-443c-a390-0323761a03fc-kube-api-access-s79fz\") pod \"success-200-isvc-db836-predictor-598457c8f9-bc6fv\" (UID: \"0b838ab0-36ca-443c-a390-0323761a03fc\") " pod="kserve-ci-e2e-test/success-200-isvc-db836-predictor-598457c8f9-bc6fv"
Jul 01 17:30:06.663378 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:06.663201    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9a6ed0aa-189f-4c19-bfee-d052a47aa0e9-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-db836-predictor-6996c59df8-gc6q8\" (UID: \"9a6ed0aa-189f-4c19-bfee-d052a47aa0e9\") " pod="kserve-ci-e2e-test/error-404-isvc-db836-predictor-6996c59df8-gc6q8"
Jul 01 17:30:06.663378 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:06.663241    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b838ab0-36ca-443c-a390-0323761a03fc-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-db836-predictor-598457c8f9-bc6fv\" (UID: \"0b838ab0-36ca-443c-a390-0323761a03fc\") " pod="kserve-ci-e2e-test/success-200-isvc-db836-predictor-598457c8f9-bc6fv"
Jul 01 17:30:06.663378 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:06.663286    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a6ed0aa-189f-4c19-bfee-d052a47aa0e9-proxy-tls\") pod \"error-404-isvc-db836-predictor-6996c59df8-gc6q8\" (UID: \"9a6ed0aa-189f-4c19-bfee-d052a47aa0e9\") " pod="kserve-ci-e2e-test/error-404-isvc-db836-predictor-6996c59df8-gc6q8"
Jul 01 17:30:06.663614 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:06.663460    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b838ab0-36ca-443c-a390-0323761a03fc-proxy-tls\") pod \"success-200-isvc-db836-predictor-598457c8f9-bc6fv\" (UID: \"0b838ab0-36ca-443c-a390-0323761a03fc\") " pod="kserve-ci-e2e-test/success-200-isvc-db836-predictor-598457c8f9-bc6fv"
Jul 01 17:30:06.663614 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:30:06.663589    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-db836-predictor-serving-cert: secret "success-200-isvc-db836-predictor-serving-cert" not found
Jul 01 17:30:06.663695 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:30:06.663647    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b838ab0-36ca-443c-a390-0323761a03fc-proxy-tls podName:0b838ab0-36ca-443c-a390-0323761a03fc nodeName:}" failed. No retries permitted until 2026-07-01 17:30:07.163625386 +0000 UTC m=+1924.960763782 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0b838ab0-36ca-443c-a390-0323761a03fc-proxy-tls") pod "success-200-isvc-db836-predictor-598457c8f9-bc6fv" (UID: "0b838ab0-36ca-443c-a390-0323761a03fc") : secret "success-200-isvc-db836-predictor-serving-cert" not found
Jul 01 17:30:06.663955 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:06.663938    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b838ab0-36ca-443c-a390-0323761a03fc-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-db836-predictor-598457c8f9-bc6fv\" (UID: \"0b838ab0-36ca-443c-a390-0323761a03fc\") " pod="kserve-ci-e2e-test/success-200-isvc-db836-predictor-598457c8f9-bc6fv"
Jul 01 17:30:06.675793 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:06.675773    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s79fz\" (UniqueName: \"kubernetes.io/projected/0b838ab0-36ca-443c-a390-0323761a03fc-kube-api-access-s79fz\") pod \"success-200-isvc-db836-predictor-598457c8f9-bc6fv\" (UID: \"0b838ab0-36ca-443c-a390-0323761a03fc\") " pod="kserve-ci-e2e-test/success-200-isvc-db836-predictor-598457c8f9-bc6fv"
Jul 01 17:30:06.764181 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:06.764147    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9a6ed0aa-189f-4c19-bfee-d052a47aa0e9-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-db836-predictor-6996c59df8-gc6q8\" (UID: \"9a6ed0aa-189f-4c19-bfee-d052a47aa0e9\") " pod="kserve-ci-e2e-test/error-404-isvc-db836-predictor-6996c59df8-gc6q8"
Jul 01 17:30:06.764391 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:06.764216    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a6ed0aa-189f-4c19-bfee-d052a47aa0e9-proxy-tls\") pod \"error-404-isvc-db836-predictor-6996c59df8-gc6q8\" (UID: \"9a6ed0aa-189f-4c19-bfee-d052a47aa0e9\") " pod="kserve-ci-e2e-test/error-404-isvc-db836-predictor-6996c59df8-gc6q8"
Jul 01 17:30:06.764391 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:06.764292    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghvqs\" (UniqueName: \"kubernetes.io/projected/9a6ed0aa-189f-4c19-bfee-d052a47aa0e9-kube-api-access-ghvqs\") pod \"error-404-isvc-db836-predictor-6996c59df8-gc6q8\" (UID: \"9a6ed0aa-189f-4c19-bfee-d052a47aa0e9\") " pod="kserve-ci-e2e-test/error-404-isvc-db836-predictor-6996c59df8-gc6q8"
Jul 01 17:30:06.764931 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:06.764907    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9a6ed0aa-189f-4c19-bfee-d052a47aa0e9-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-db836-predictor-6996c59df8-gc6q8\" (UID: \"9a6ed0aa-189f-4c19-bfee-d052a47aa0e9\") " pod="kserve-ci-e2e-test/error-404-isvc-db836-predictor-6996c59df8-gc6q8"
Jul 01 17:30:06.766856 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:06.766837    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a6ed0aa-189f-4c19-bfee-d052a47aa0e9-proxy-tls\") pod \"error-404-isvc-db836-predictor-6996c59df8-gc6q8\" (UID: \"9a6ed0aa-189f-4c19-bfee-d052a47aa0e9\") " pod="kserve-ci-e2e-test/error-404-isvc-db836-predictor-6996c59df8-gc6q8"
Jul 01 17:30:06.773450 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:06.773421    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghvqs\" (UniqueName: \"kubernetes.io/projected/9a6ed0aa-189f-4c19-bfee-d052a47aa0e9-kube-api-access-ghvqs\") pod \"error-404-isvc-db836-predictor-6996c59df8-gc6q8\" (UID: \"9a6ed0aa-189f-4c19-bfee-d052a47aa0e9\") " pod="kserve-ci-e2e-test/error-404-isvc-db836-predictor-6996c59df8-gc6q8"
Jul 01 17:30:06.920723 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:06.920637    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-db836-predictor-6996c59df8-gc6q8"
Jul 01 17:30:07.062184 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:07.062151    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-db836-predictor-6996c59df8-gc6q8"]
Jul 01 17:30:07.064429 ip-10-0-137-180 kubenswrapper[2563]: W0701 17:30:07.064405    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a6ed0aa_189f_4c19_bfee_d052a47aa0e9.slice/crio-6eece7c8fb4fbc268821c83a612ae831ff9fc8031500b356ecc07b9fdc1bc2d2 WatchSource:0}: Error finding container 6eece7c8fb4fbc268821c83a612ae831ff9fc8031500b356ecc07b9fdc1bc2d2: Status 404 returned error can't find the container with id 6eece7c8fb4fbc268821c83a612ae831ff9fc8031500b356ecc07b9fdc1bc2d2
Jul 01 17:30:07.166870 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:07.166840    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b838ab0-36ca-443c-a390-0323761a03fc-proxy-tls\") pod \"success-200-isvc-db836-predictor-598457c8f9-bc6fv\" (UID: \"0b838ab0-36ca-443c-a390-0323761a03fc\") " pod="kserve-ci-e2e-test/success-200-isvc-db836-predictor-598457c8f9-bc6fv"
Jul 01 17:30:07.169148 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:07.169126    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b838ab0-36ca-443c-a390-0323761a03fc-proxy-tls\") pod \"success-200-isvc-db836-predictor-598457c8f9-bc6fv\" (UID: \"0b838ab0-36ca-443c-a390-0323761a03fc\") " pod="kserve-ci-e2e-test/success-200-isvc-db836-predictor-598457c8f9-bc6fv"
Jul 01 17:30:07.297907 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:07.297874    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4517b-predictor-64b9b9f5bc-9pcrm" podUID="834629ee-e276-4d11-934b-2b6a7179ee66" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 01 17:30:07.332118 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:07.332081    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:30:07.405248 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:07.405214    2563 generic.go:356] "Generic (PLEG): container finished" podID="5bb651ab-06a8-4dbc-812f-1eb6ac74f174" containerID="39069cb5512cbd7334cbc4d70984d4493cc4eace9eff5526dae1f2087a92482f" exitCode=2
Jul 01 17:30:07.405464 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:07.405284    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6460c-predictor-5774675f9-x8h72" event={"ID":"5bb651ab-06a8-4dbc-812f-1eb6ac74f174","Type":"ContainerDied","Data":"39069cb5512cbd7334cbc4d70984d4493cc4eace9eff5526dae1f2087a92482f"}
Jul 01 17:30:07.406979 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:07.406954    2563 generic.go:356] "Generic (PLEG): container finished" podID="c6b534f1-c10d-494c-8547-05f4a1073b40" containerID="480dde0a78af44494f13050f968554628cc80278be8f15cec90da3852c52fe92" exitCode=2
Jul 01 17:30:07.407100 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:07.407041    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6460c-predictor-5d5d94dfb6-btq8d" event={"ID":"c6b534f1-c10d-494c-8547-05f4a1073b40","Type":"ContainerDied","Data":"480dde0a78af44494f13050f968554628cc80278be8f15cec90da3852c52fe92"}
Jul 01 17:30:07.408656 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:07.408625    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-db836-predictor-6996c59df8-gc6q8" event={"ID":"9a6ed0aa-189f-4c19-bfee-d052a47aa0e9","Type":"ContainerStarted","Data":"adb088884152a4fda39fb8cde0a146a2d3139e38506b3c87c3acf00c616253fd"}
Jul 01 17:30:07.408776 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:07.408685    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-db836-predictor-6996c59df8-gc6q8" event={"ID":"9a6ed0aa-189f-4c19-bfee-d052a47aa0e9","Type":"ContainerStarted","Data":"dda22b062e8a5075d2ad285e947298bfd9c2ec6f4d00f77161363b8bc629c2a0"}
Jul 01 17:30:07.408776 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:07.408725    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-db836-predictor-6996c59df8-gc6q8" event={"ID":"9a6ed0aa-189f-4c19-bfee-d052a47aa0e9","Type":"ContainerStarted","Data":"6eece7c8fb4fbc268821c83a612ae831ff9fc8031500b356ecc07b9fdc1bc2d2"}
Jul 01 17:30:07.409788 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:07.409762    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-db836-predictor-6996c59df8-gc6q8"
Jul 01 17:30:07.409884 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:07.409817    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-db836-predictor-6996c59df8-gc6q8"
Jul 01 17:30:07.411533 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:07.411498    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-db836-predictor-6996c59df8-gc6q8" podUID="9a6ed0aa-189f-4c19-bfee-d052a47aa0e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 01 17:30:07.420713 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:07.420677    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-db836-predictor-598457c8f9-bc6fv"
Jul 01 17:30:07.429185 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:07.429100    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-db836-predictor-6996c59df8-gc6q8" podStartSLOduration=1.429082457 podStartE2EDuration="1.429082457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:30:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:30:07.428134562 +0000 UTC m=+1925.225272965" watchObservedRunningTime="2026-07-01 17:30:07.429082457 +0000 UTC m=+1925.226220858"
Jul 01 17:30:07.514564 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:07.514534    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:30:07.563392 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:07.563284    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-db836-predictor-598457c8f9-bc6fv"]
Jul 01 17:30:07.565562 ip-10-0-137-180 kubenswrapper[2563]: W0701 17:30:07.565534    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b838ab0_36ca_443c_a390_0323761a03fc.slice/crio-051f3f246a86658332d98b36320092693536048c10bad5a40b87334f9cb9e141 WatchSource:0}: Error finding container 051f3f246a86658332d98b36320092693536048c10bad5a40b87334f9cb9e141: Status 404 returned error can't find the container with id 051f3f246a86658332d98b36320092693536048c10bad5a40b87334f9cb9e141
Jul 01 17:30:08.418173 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:08.418127    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-db836-predictor-598457c8f9-bc6fv" event={"ID":"0b838ab0-36ca-443c-a390-0323761a03fc","Type":"ContainerStarted","Data":"9c9d4d7f6001b89e3792303b95f304daa74c6e55b3d2754001c585c5472d66ed"}
Jul 01 17:30:08.418669 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:08.418199    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-db836-predictor-598457c8f9-bc6fv" event={"ID":"0b838ab0-36ca-443c-a390-0323761a03fc","Type":"ContainerStarted","Data":"2069fe2e4ea3a2187b93817d9f0dfd766087ae84ca465533dc07dcffce19afa3"}
Jul 01 17:30:08.418669 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:08.418238    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-db836-predictor-598457c8f9-bc6fv" event={"ID":"0b838ab0-36ca-443c-a390-0323761a03fc","Type":"ContainerStarted","Data":"051f3f246a86658332d98b36320092693536048c10bad5a40b87334f9cb9e141"}
Jul 01 17:30:08.418669 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:08.418520    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-db836-predictor-598457c8f9-bc6fv"
Jul 01 17:30:08.418669 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:08.418532    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-db836-predictor-6996c59df8-gc6q8" podUID="9a6ed0aa-189f-4c19-bfee-d052a47aa0e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 01 17:30:08.418669 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:08.418579    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-db836-predictor-598457c8f9-bc6fv"
Jul 01 17:30:08.420008 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:08.419987    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-db836-predictor-598457c8f9-bc6fv" podUID="0b838ab0-36ca-443c-a390-0323761a03fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 01 17:30:08.438161 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:08.438115    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-db836-predictor-598457c8f9-bc6fv" podStartSLOduration=2.4380991610000002 podStartE2EDuration="2.438099161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:30:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:30:08.436461635 +0000 UTC m=+1926.233600055" watchObservedRunningTime="2026-07-01 17:30:08.438099161 +0000 UTC m=+1926.235237561"
Jul 01 17:30:09.308194 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:09.308158    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4517b-predictor-677799f794-kj8t9" podUID="73bda8d8-398f-46d6-a526-be667395fad7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 01 17:30:09.427213 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:09.427169    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-db836-predictor-598457c8f9-bc6fv" podUID="0b838ab0-36ca-443c-a390-0323761a03fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 01 17:30:09.427607 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:09.427227    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-db836-predictor-6996c59df8-gc6q8" podUID="9a6ed0aa-189f-4c19-bfee-d052a47aa0e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 01 17:30:10.093700 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:10.093675    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-6460c-predictor-5774675f9-x8h72"
Jul 01 17:30:10.181904 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:10.181813    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6460c-predictor-5d5d94dfb6-btq8d" podUID="c6b534f1-c10d-494c-8547-05f4a1073b40" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.37:8643/healthz\": dial tcp 10.132.0.37:8643: connect: connection refused"
Jul 01 17:30:10.186833 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:10.186809    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6460c-predictor-5d5d94dfb6-btq8d" podUID="c6b534f1-c10d-494c-8547-05f4a1073b40" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 01 17:30:10.192087 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:10.192069    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5bb651ab-06a8-4dbc-812f-1eb6ac74f174-kube-rbac-proxy-sar-config\") pod \"5bb651ab-06a8-4dbc-812f-1eb6ac74f174\" (UID: \"5bb651ab-06a8-4dbc-812f-1eb6ac74f174\") "
Jul 01 17:30:10.192152 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:10.192107    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q8xc2\" (UniqueName: \"kubernetes.io/projected/5bb651ab-06a8-4dbc-812f-1eb6ac74f174-kube-api-access-q8xc2\") pod \"5bb651ab-06a8-4dbc-812f-1eb6ac74f174\" (UID: \"5bb651ab-06a8-4dbc-812f-1eb6ac74f174\") "
Jul 01 17:30:10.192152 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:10.192129    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5bb651ab-06a8-4dbc-812f-1eb6ac74f174-proxy-tls\") pod \"5bb651ab-06a8-4dbc-812f-1eb6ac74f174\" (UID: \"5bb651ab-06a8-4dbc-812f-1eb6ac74f174\") "
Jul 01 17:30:10.192424 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:10.192402    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bb651ab-06a8-4dbc-812f-1eb6ac74f174-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5bb651ab-06a8-4dbc-812f-1eb6ac74f174" (UID: "5bb651ab-06a8-4dbc-812f-1eb6ac74f174"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:30:10.194252 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:10.194225    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bb651ab-06a8-4dbc-812f-1eb6ac74f174-kube-api-access-q8xc2" (OuterVolumeSpecName: "kube-api-access-q8xc2") pod "5bb651ab-06a8-4dbc-812f-1eb6ac74f174" (UID: "5bb651ab-06a8-4dbc-812f-1eb6ac74f174"). InnerVolumeSpecName "kube-api-access-q8xc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 17:30:10.194252 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:10.194230    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bb651ab-06a8-4dbc-812f-1eb6ac74f174-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5bb651ab-06a8-4dbc-812f-1eb6ac74f174" (UID: "5bb651ab-06a8-4dbc-812f-1eb6ac74f174"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 17:30:10.293780 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:10.293746    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5bb651ab-06a8-4dbc-812f-1eb6ac74f174-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:30:10.293780 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:10.293775    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8xc2\" (UniqueName: \"kubernetes.io/projected/5bb651ab-06a8-4dbc-812f-1eb6ac74f174-kube-api-access-q8xc2\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:30:10.293780 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:10.293787    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5bb651ab-06a8-4dbc-812f-1eb6ac74f174-proxy-tls\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:30:10.419674 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:10.419648    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-6460c-predictor-5d5d94dfb6-btq8d"
Jul 01 17:30:10.432220 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:10.432152    2563 generic.go:356] "Generic (PLEG): container finished" podID="c6b534f1-c10d-494c-8547-05f4a1073b40" containerID="42eff41b2c0479a41b1f9f28f136bf2093444e12ecc5932aec535790da72479d" exitCode=0
Jul 01 17:30:10.432220 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:10.432206    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-6460c-predictor-5d5d94dfb6-btq8d"
Jul 01 17:30:10.432681 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:10.432239    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6460c-predictor-5d5d94dfb6-btq8d" event={"ID":"c6b534f1-c10d-494c-8547-05f4a1073b40","Type":"ContainerDied","Data":"42eff41b2c0479a41b1f9f28f136bf2093444e12ecc5932aec535790da72479d"}
Jul 01 17:30:10.432681 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:10.432339    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6460c-predictor-5d5d94dfb6-btq8d" event={"ID":"c6b534f1-c10d-494c-8547-05f4a1073b40","Type":"ContainerDied","Data":"e1e77a18390cef67ce95efa0928755b91669d939e6902cb9aba95bc12522aa6a"}
Jul 01 17:30:10.432681 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:10.432374    2563 scope.go:117] "RemoveContainer" containerID="480dde0a78af44494f13050f968554628cc80278be8f15cec90da3852c52fe92"
Jul 01 17:30:10.433843 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:10.433824    2563 generic.go:356] "Generic (PLEG): container finished" podID="5bb651ab-06a8-4dbc-812f-1eb6ac74f174" containerID="153961105a76fbfc43c3457a4fa2fe9bc3c9e202fc2691149f798539103ac196" exitCode=0
Jul 01 17:30:10.433960 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:10.433870    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-6460c-predictor-5774675f9-x8h72"
Jul 01 17:30:10.433960 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:10.433908    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6460c-predictor-5774675f9-x8h72" event={"ID":"5bb651ab-06a8-4dbc-812f-1eb6ac74f174","Type":"ContainerDied","Data":"153961105a76fbfc43c3457a4fa2fe9bc3c9e202fc2691149f798539103ac196"}
Jul 01 17:30:10.433960 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:10.433954    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6460c-predictor-5774675f9-x8h72" event={"ID":"5bb651ab-06a8-4dbc-812f-1eb6ac74f174","Type":"ContainerDied","Data":"84d391b5e7913b20e251734366832e6400a1a4f6b3798591266bbaf4f5e05682"}
Jul 01 17:30:10.434396 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:10.434367    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-db836-predictor-598457c8f9-bc6fv" podUID="0b838ab0-36ca-443c-a390-0323761a03fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 01 17:30:10.441302 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:10.441277    2563 scope.go:117] "RemoveContainer" containerID="42eff41b2c0479a41b1f9f28f136bf2093444e12ecc5932aec535790da72479d"
Jul 01 17:30:10.450282 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:10.450257    2563 scope.go:117] "RemoveContainer" containerID="480dde0a78af44494f13050f968554628cc80278be8f15cec90da3852c52fe92"
Jul 01 17:30:10.450540 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:30:10.450521    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"480dde0a78af44494f13050f968554628cc80278be8f15cec90da3852c52fe92\": container with ID starting with 480dde0a78af44494f13050f968554628cc80278be8f15cec90da3852c52fe92 not found: ID does not exist" containerID="480dde0a78af44494f13050f968554628cc80278be8f15cec90da3852c52fe92"
Jul 01 17:30:10.450598 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:10.450547    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"480dde0a78af44494f13050f968554628cc80278be8f15cec90da3852c52fe92"} err="failed to get container status \"480dde0a78af44494f13050f968554628cc80278be8f15cec90da3852c52fe92\": rpc error: code = NotFound desc = could not find container \"480dde0a78af44494f13050f968554628cc80278be8f15cec90da3852c52fe92\": container with ID starting with 480dde0a78af44494f13050f968554628cc80278be8f15cec90da3852c52fe92 not found: ID does not exist"
Jul 01 17:30:10.450598 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:10.450562    2563 scope.go:117] "RemoveContainer" containerID="42eff41b2c0479a41b1f9f28f136bf2093444e12ecc5932aec535790da72479d"
Jul 01 17:30:10.450825 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:30:10.450808    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42eff41b2c0479a41b1f9f28f136bf2093444e12ecc5932aec535790da72479d\": container with ID starting with 42eff41b2c0479a41b1f9f28f136bf2093444e12ecc5932aec535790da72479d not found: ID does not exist" containerID="42eff41b2c0479a41b1f9f28f136bf2093444e12ecc5932aec535790da72479d"
Jul 01 17:30:10.450874 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:10.450832    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42eff41b2c0479a41b1f9f28f136bf2093444e12ecc5932aec535790da72479d"} err="failed to get container status \"42eff41b2c0479a41b1f9f28f136bf2093444e12ecc5932aec535790da72479d\": rpc error: code = NotFound desc = could not find container \"42eff41b2c0479a41b1f9f28f136bf2093444e12ecc5932aec535790da72479d\": container with ID starting with 42eff41b2c0479a41b1f9f28f136bf2093444e12ecc5932aec535790da72479d not found: ID does not exist"
Jul 01 17:30:10.450874 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:10.450847    2563 scope.go:117] "RemoveContainer" containerID="39069cb5512cbd7334cbc4d70984d4493cc4eace9eff5526dae1f2087a92482f"
Jul 01 17:30:10.458441 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:10.458419    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6460c-predictor-5774675f9-x8h72"]
Jul 01 17:30:10.459308 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:10.459300    2563 scope.go:117] "RemoveContainer" containerID="153961105a76fbfc43c3457a4fa2fe9bc3c9e202fc2691149f798539103ac196"
Jul 01 17:30:10.463296 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:10.463268    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6460c-predictor-5774675f9-x8h72"]
Jul 01 17:30:10.467450 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:10.467433    2563 scope.go:117] "RemoveContainer" containerID="39069cb5512cbd7334cbc4d70984d4493cc4eace9eff5526dae1f2087a92482f"
Jul 01 17:30:10.467741 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:30:10.467712    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39069cb5512cbd7334cbc4d70984d4493cc4eace9eff5526dae1f2087a92482f\": container with ID starting with 39069cb5512cbd7334cbc4d70984d4493cc4eace9eff5526dae1f2087a92482f not found: ID does not exist" containerID="39069cb5512cbd7334cbc4d70984d4493cc4eace9eff5526dae1f2087a92482f"
Jul 01 17:30:10.467806 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:10.467743    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39069cb5512cbd7334cbc4d70984d4493cc4eace9eff5526dae1f2087a92482f"} err="failed to get container status \"39069cb5512cbd7334cbc4d70984d4493cc4eace9eff5526dae1f2087a92482f\": rpc error: code = NotFound desc = could not find container \"39069cb5512cbd7334cbc4d70984d4493cc4eace9eff5526dae1f2087a92482f\": container with ID starting with 39069cb5512cbd7334cbc4d70984d4493cc4eace9eff5526dae1f2087a92482f not found: ID does not exist"
Jul 01 17:30:10.467806 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:10.467761    2563 scope.go:117] "RemoveContainer" containerID="153961105a76fbfc43c3457a4fa2fe9bc3c9e202fc2691149f798539103ac196"
Jul 01 17:30:10.467997 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:30:10.467983    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"153961105a76fbfc43c3457a4fa2fe9bc3c9e202fc2691149f798539103ac196\": container with ID starting with 153961105a76fbfc43c3457a4fa2fe9bc3c9e202fc2691149f798539103ac196 not found: ID does not exist" containerID="153961105a76fbfc43c3457a4fa2fe9bc3c9e202fc2691149f798539103ac196"
Jul 01 17:30:10.468037 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:10.468009    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"153961105a76fbfc43c3457a4fa2fe9bc3c9e202fc2691149f798539103ac196"} err="failed to get container status \"153961105a76fbfc43c3457a4fa2fe9bc3c9e202fc2691149f798539103ac196\": rpc error: code = NotFound desc = could not find container \"153961105a76fbfc43c3457a4fa2fe9bc3c9e202fc2691149f798539103ac196\": container with ID starting with 153961105a76fbfc43c3457a4fa2fe9bc3c9e202fc2691149f798539103ac196 not found: ID does not exist"
Jul 01 17:30:10.606103 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:10.606069    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6b534f1-c10d-494c-8547-05f4a1073b40-kube-rbac-proxy-sar-config\") pod \"c6b534f1-c10d-494c-8547-05f4a1073b40\" (UID: \"c6b534f1-c10d-494c-8547-05f4a1073b40\") "
Jul 01 17:30:10.606286 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:10.606108    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6b534f1-c10d-494c-8547-05f4a1073b40-proxy-tls\") pod \"c6b534f1-c10d-494c-8547-05f4a1073b40\" (UID: \"c6b534f1-c10d-494c-8547-05f4a1073b40\") "
Jul 01 17:30:10.606286 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:10.606136    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w7jrn\" (UniqueName: \"kubernetes.io/projected/c6b534f1-c10d-494c-8547-05f4a1073b40-kube-api-access-w7jrn\") pod \"c6b534f1-c10d-494c-8547-05f4a1073b40\" (UID: \"c6b534f1-c10d-494c-8547-05f4a1073b40\") "
Jul 01 17:30:10.606539 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:10.606500    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6b534f1-c10d-494c-8547-05f4a1073b40-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c6b534f1-c10d-494c-8547-05f4a1073b40" (UID: "c6b534f1-c10d-494c-8547-05f4a1073b40"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:30:10.608306 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:10.608285    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6b534f1-c10d-494c-8547-05f4a1073b40-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c6b534f1-c10d-494c-8547-05f4a1073b40" (UID: "c6b534f1-c10d-494c-8547-05f4a1073b40"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 17:30:10.608423 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:10.608309    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6b534f1-c10d-494c-8547-05f4a1073b40-kube-api-access-w7jrn" (OuterVolumeSpecName: "kube-api-access-w7jrn") pod "c6b534f1-c10d-494c-8547-05f4a1073b40" (UID: "c6b534f1-c10d-494c-8547-05f4a1073b40"). InnerVolumeSpecName "kube-api-access-w7jrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 17:30:10.697321 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:10.697240    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5bb651ab-06a8-4dbc-812f-1eb6ac74f174" path="/var/lib/kubelet/pods/5bb651ab-06a8-4dbc-812f-1eb6ac74f174/volumes"
Jul 01 17:30:10.706570 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:10.706544    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6b534f1-c10d-494c-8547-05f4a1073b40-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:30:10.706570 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:10.706571    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6b534f1-c10d-494c-8547-05f4a1073b40-proxy-tls\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:30:10.706763 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:10.706583    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7jrn\" (UniqueName: \"kubernetes.io/projected/c6b534f1-c10d-494c-8547-05f4a1073b40-kube-api-access-w7jrn\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:30:10.750171 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:10.750148    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6460c-predictor-5d5d94dfb6-btq8d"]
Jul 01 17:30:10.754479 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:10.754457    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6460c-predictor-5d5d94dfb6-btq8d"]
Jul 01 17:30:12.698077 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:12.698043    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6b534f1-c10d-494c-8547-05f4a1073b40" path="/var/lib/kubelet/pods/c6b534f1-c10d-494c-8547-05f4a1073b40/volumes"
Jul 01 17:30:14.431853 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:14.431823    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-db836-predictor-6996c59df8-gc6q8"
Jul 01 17:30:14.432330 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:14.432244    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-db836-predictor-6996c59df8-gc6q8" podUID="9a6ed0aa-189f-4c19-bfee-d052a47aa0e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 01 17:30:15.440129 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:15.440100    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-db836-predictor-598457c8f9-bc6fv"
Jul 01 17:30:15.440553 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:15.440526    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-db836-predictor-598457c8f9-bc6fv" podUID="0b838ab0-36ca-443c-a390-0323761a03fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 01 17:30:17.298131 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:17.298092    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4517b-predictor-64b9b9f5bc-9pcrm" podUID="834629ee-e276-4d11-934b-2b6a7179ee66" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 01 17:30:19.308582 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:19.308540    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4517b-predictor-677799f794-kj8t9" podUID="73bda8d8-398f-46d6-a526-be667395fad7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 01 17:30:24.432527 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:24.432488    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-db836-predictor-6996c59df8-gc6q8" podUID="9a6ed0aa-189f-4c19-bfee-d052a47aa0e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 01 17:30:25.441279 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:25.441231    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-db836-predictor-598457c8f9-bc6fv" podUID="0b838ab0-36ca-443c-a390-0323761a03fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 01 17:30:27.297933 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:27.297899    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4517b-predictor-64b9b9f5bc-9pcrm" podUID="834629ee-e276-4d11-934b-2b6a7179ee66" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 01 17:30:29.307809 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:29.307761    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4517b-predictor-677799f794-kj8t9" podUID="73bda8d8-398f-46d6-a526-be667395fad7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 01 17:30:34.432904 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:34.432861    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-db836-predictor-6996c59df8-gc6q8" podUID="9a6ed0aa-189f-4c19-bfee-d052a47aa0e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 01 17:30:35.441021 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:35.440982    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-db836-predictor-598457c8f9-bc6fv" podUID="0b838ab0-36ca-443c-a390-0323761a03fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 01 17:30:37.298514 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:37.298477    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-4517b-predictor-64b9b9f5bc-9pcrm"
Jul 01 17:30:39.308528 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:39.308496    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-4517b-predictor-677799f794-kj8t9"
Jul 01 17:30:44.432296 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:44.432245    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-db836-predictor-6996c59df8-gc6q8" podUID="9a6ed0aa-189f-4c19-bfee-d052a47aa0e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 01 17:30:45.440618 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:45.440583    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-db836-predictor-598457c8f9-bc6fv" podUID="0b838ab0-36ca-443c-a390-0323761a03fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 01 17:30:50.727466 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:50.727436    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:30:51.378445 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:51.378410    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:30:54.432880 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:54.432801    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-db836-predictor-6996c59df8-gc6q8" podUID="9a6ed0aa-189f-4c19-bfee-d052a47aa0e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 01 17:30:55.442523 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:30:55.442496    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-db836-predictor-598457c8f9-bc6fv"
Jul 01 17:31:00.800804 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:00.800774    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:31:00.932601 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:00.932558    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4517b-predictor-64b9b9f5bc-9pcrm"]
Jul 01 17:31:00.932857 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:00.932812    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-4517b-predictor-64b9b9f5bc-9pcrm" podUID="834629ee-e276-4d11-934b-2b6a7179ee66" containerName="kserve-container" containerID="cri-o://2a22ccc9d7a74de3ed9f6307c617fb5ad4c9a58d9d5173bf45e3ca8635b03002" gracePeriod=30
Jul 01 17:31:00.933001 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:00.932864    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-4517b-predictor-64b9b9f5bc-9pcrm" podUID="834629ee-e276-4d11-934b-2b6a7179ee66" containerName="kube-rbac-proxy" containerID="cri-o://938988ab4bedc0e898ad8ecac1c8f05472a3e6ed1344958b8626959144c3db6e" gracePeriod=30
Jul 01 17:31:00.935184 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:00.935162    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bbf0f-predictor-6d7cf8c5c-bc4gn"]
Jul 01 17:31:00.935523 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:00.935509    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bb651ab-06a8-4dbc-812f-1eb6ac74f174" containerName="kserve-container"
Jul 01 17:31:00.935574 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:00.935525    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bb651ab-06a8-4dbc-812f-1eb6ac74f174" containerName="kserve-container"
Jul 01 17:31:00.935574 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:00.935539    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bb651ab-06a8-4dbc-812f-1eb6ac74f174" containerName="kube-rbac-proxy"
Jul 01 17:31:00.935574 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:00.935545    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bb651ab-06a8-4dbc-812f-1eb6ac74f174" containerName="kube-rbac-proxy"
Jul 01 17:31:00.935574 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:00.935560    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6b534f1-c10d-494c-8547-05f4a1073b40" containerName="kube-rbac-proxy"
Jul 01 17:31:00.935574 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:00.935566    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6b534f1-c10d-494c-8547-05f4a1073b40" containerName="kube-rbac-proxy"
Jul 01 17:31:00.935737 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:00.935580    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6b534f1-c10d-494c-8547-05f4a1073b40" containerName="kserve-container"
Jul 01 17:31:00.935737 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:00.935587    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6b534f1-c10d-494c-8547-05f4a1073b40" containerName="kserve-container"
Jul 01 17:31:00.935737 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:00.935641    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6b534f1-c10d-494c-8547-05f4a1073b40" containerName="kube-rbac-proxy"
Jul 01 17:31:00.935737 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:00.935650    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bb651ab-06a8-4dbc-812f-1eb6ac74f174" containerName="kube-rbac-proxy"
Jul 01 17:31:00.935737 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:00.935658    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bb651ab-06a8-4dbc-812f-1eb6ac74f174" containerName="kserve-container"
Jul 01 17:31:00.935737 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:00.935663    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6b534f1-c10d-494c-8547-05f4a1073b40" containerName="kserve-container"
Jul 01 17:31:00.939382 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:00.939087    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-bbf0f-predictor-6d7cf8c5c-bc4gn"
Jul 01 17:31:00.943712 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:00.943687    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-bbf0f-predictor-serving-cert\""
Jul 01 17:31:00.944048 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:00.944033    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-bbf0f-kube-rbac-proxy-sar-config\""
Jul 01 17:31:00.957860 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:00.957830    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bbf0f-predictor-6d7cf8c5c-bc4gn"]
Jul 01 17:31:01.035126 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:01.035088    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx9bt\" (UniqueName: \"kubernetes.io/projected/416f5e13-7be6-4ebf-a032-2295aca3c479-kube-api-access-gx9bt\") pod \"success-200-isvc-bbf0f-predictor-6d7cf8c5c-bc4gn\" (UID: \"416f5e13-7be6-4ebf-a032-2295aca3c479\") " pod="kserve-ci-e2e-test/success-200-isvc-bbf0f-predictor-6d7cf8c5c-bc4gn"
Jul 01 17:31:01.035294 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:01.035156    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/416f5e13-7be6-4ebf-a032-2295aca3c479-proxy-tls\") pod \"success-200-isvc-bbf0f-predictor-6d7cf8c5c-bc4gn\" (UID: \"416f5e13-7be6-4ebf-a032-2295aca3c479\") " pod="kserve-ci-e2e-test/success-200-isvc-bbf0f-predictor-6d7cf8c5c-bc4gn"
Jul 01 17:31:01.035294 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:01.035210    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/416f5e13-7be6-4ebf-a032-2295aca3c479-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-bbf0f-predictor-6d7cf8c5c-bc4gn\" (UID: \"416f5e13-7be6-4ebf-a032-2295aca3c479\") " pod="kserve-ci-e2e-test/success-200-isvc-bbf0f-predictor-6d7cf8c5c-bc4gn"
Jul 01 17:31:01.062496 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:01.062458    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bbf0f-predictor-85ff6cdb4-jzkmn"]
Jul 01 17:31:01.065864 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:01.065847    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-bbf0f-predictor-85ff6cdb4-jzkmn"
Jul 01 17:31:01.068664 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:01.068636    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-bbf0f-predictor-serving-cert\""
Jul 01 17:31:01.068849 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:01.068829    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-bbf0f-kube-rbac-proxy-sar-config\""
Jul 01 17:31:01.084458 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:01.084428    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bbf0f-predictor-85ff6cdb4-jzkmn"]
Jul 01 17:31:01.132612 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:01.132571    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4517b-predictor-677799f794-kj8t9"]
Jul 01 17:31:01.132827 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:01.132800    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-4517b-predictor-677799f794-kj8t9" podUID="73bda8d8-398f-46d6-a526-be667395fad7" containerName="kserve-container" containerID="cri-o://b88c8d76f8e3a0674ad8135a026550dd7b1e5ba8ccdd40202d2db514407fd16e" gracePeriod=30
Jul 01 17:31:01.132892 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:01.132862    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-4517b-predictor-677799f794-kj8t9" podUID="73bda8d8-398f-46d6-a526-be667395fad7" containerName="kube-rbac-proxy" containerID="cri-o://60de77b307e4f9ee24a4428d99b45969256646b7030b7788b13446682d8ed8d5" gracePeriod=30
Jul 01 17:31:01.136442 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:01.136408    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/416f5e13-7be6-4ebf-a032-2295aca3c479-proxy-tls\") pod \"success-200-isvc-bbf0f-predictor-6d7cf8c5c-bc4gn\" (UID: \"416f5e13-7be6-4ebf-a032-2295aca3c479\") " pod="kserve-ci-e2e-test/success-200-isvc-bbf0f-predictor-6d7cf8c5c-bc4gn"
Jul 01 17:31:01.136604 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:01.136477    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/416f5e13-7be6-4ebf-a032-2295aca3c479-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-bbf0f-predictor-6d7cf8c5c-bc4gn\" (UID: \"416f5e13-7be6-4ebf-a032-2295aca3c479\") " pod="kserve-ci-e2e-test/success-200-isvc-bbf0f-predictor-6d7cf8c5c-bc4gn"
Jul 01 17:31:01.136604 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:01.136550    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx9bt\" (UniqueName: \"kubernetes.io/projected/416f5e13-7be6-4ebf-a032-2295aca3c479-kube-api-access-gx9bt\") pod \"success-200-isvc-bbf0f-predictor-6d7cf8c5c-bc4gn\" (UID: \"416f5e13-7be6-4ebf-a032-2295aca3c479\") " pod="kserve-ci-e2e-test/success-200-isvc-bbf0f-predictor-6d7cf8c5c-bc4gn"
Jul 01 17:31:01.136604 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:31:01.136576    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-bbf0f-predictor-serving-cert: secret "success-200-isvc-bbf0f-predictor-serving-cert" not found
Jul 01 17:31:01.136750 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:31:01.136649    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/416f5e13-7be6-4ebf-a032-2295aca3c479-proxy-tls podName:416f5e13-7be6-4ebf-a032-2295aca3c479 nodeName:}" failed. No retries permitted until 2026-07-01 17:31:01.636628152 +0000 UTC m=+1979.433766536 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/416f5e13-7be6-4ebf-a032-2295aca3c479-proxy-tls") pod "success-200-isvc-bbf0f-predictor-6d7cf8c5c-bc4gn" (UID: "416f5e13-7be6-4ebf-a032-2295aca3c479") : secret "success-200-isvc-bbf0f-predictor-serving-cert" not found
Jul 01 17:31:01.137224 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:01.137197    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/416f5e13-7be6-4ebf-a032-2295aca3c479-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-bbf0f-predictor-6d7cf8c5c-bc4gn\" (UID: \"416f5e13-7be6-4ebf-a032-2295aca3c479\") " pod="kserve-ci-e2e-test/success-200-isvc-bbf0f-predictor-6d7cf8c5c-bc4gn"
Jul 01 17:31:01.146469 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:01.146446    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx9bt\" (UniqueName: \"kubernetes.io/projected/416f5e13-7be6-4ebf-a032-2295aca3c479-kube-api-access-gx9bt\") pod \"success-200-isvc-bbf0f-predictor-6d7cf8c5c-bc4gn\" (UID: \"416f5e13-7be6-4ebf-a032-2295aca3c479\") " pod="kserve-ci-e2e-test/success-200-isvc-bbf0f-predictor-6d7cf8c5c-bc4gn"
Jul 01 17:31:01.237294 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:01.237242    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3121a651-3759-41c0-8363-d5cf6fbb991a-proxy-tls\") pod \"error-404-isvc-bbf0f-predictor-85ff6cdb4-jzkmn\" (UID: \"3121a651-3759-41c0-8363-d5cf6fbb991a\") " pod="kserve-ci-e2e-test/error-404-isvc-bbf0f-predictor-85ff6cdb4-jzkmn"
Jul 01 17:31:01.237508 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:01.237323    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb66d\" (UniqueName: \"kubernetes.io/projected/3121a651-3759-41c0-8363-d5cf6fbb991a-kube-api-access-nb66d\") pod \"error-404-isvc-bbf0f-predictor-85ff6cdb4-jzkmn\" (UID: \"3121a651-3759-41c0-8363-d5cf6fbb991a\") " pod="kserve-ci-e2e-test/error-404-isvc-bbf0f-predictor-85ff6cdb4-jzkmn"
Jul 01 17:31:01.237508 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:01.237378    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3121a651-3759-41c0-8363-d5cf6fbb991a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-bbf0f-predictor-85ff6cdb4-jzkmn\" (UID: \"3121a651-3759-41c0-8363-d5cf6fbb991a\") " pod="kserve-ci-e2e-test/error-404-isvc-bbf0f-predictor-85ff6cdb4-jzkmn"
Jul 01 17:31:01.338532 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:01.338440    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3121a651-3759-41c0-8363-d5cf6fbb991a-proxy-tls\") pod \"error-404-isvc-bbf0f-predictor-85ff6cdb4-jzkmn\" (UID: \"3121a651-3759-41c0-8363-d5cf6fbb991a\") " pod="kserve-ci-e2e-test/error-404-isvc-bbf0f-predictor-85ff6cdb4-jzkmn"
Jul 01 17:31:01.338532 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:01.338489    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb66d\" (UniqueName: \"kubernetes.io/projected/3121a651-3759-41c0-8363-d5cf6fbb991a-kube-api-access-nb66d\") pod \"error-404-isvc-bbf0f-predictor-85ff6cdb4-jzkmn\" (UID: \"3121a651-3759-41c0-8363-d5cf6fbb991a\") " pod="kserve-ci-e2e-test/error-404-isvc-bbf0f-predictor-85ff6cdb4-jzkmn"
Jul 01 17:31:01.338532 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:01.338516    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3121a651-3759-41c0-8363-d5cf6fbb991a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-bbf0f-predictor-85ff6cdb4-jzkmn\" (UID: \"3121a651-3759-41c0-8363-d5cf6fbb991a\") " pod="kserve-ci-e2e-test/error-404-isvc-bbf0f-predictor-85ff6cdb4-jzkmn"
Jul 01 17:31:01.338808 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:31:01.338617    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-bbf0f-predictor-serving-cert: secret "error-404-isvc-bbf0f-predictor-serving-cert" not found
Jul 01 17:31:01.338808 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:31:01.338712    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3121a651-3759-41c0-8363-d5cf6fbb991a-proxy-tls podName:3121a651-3759-41c0-8363-d5cf6fbb991a nodeName:}" failed. No retries permitted until 2026-07-01 17:31:01.838681152 +0000 UTC m=+1979.635819539 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3121a651-3759-41c0-8363-d5cf6fbb991a-proxy-tls") pod "error-404-isvc-bbf0f-predictor-85ff6cdb4-jzkmn" (UID: "3121a651-3759-41c0-8363-d5cf6fbb991a") : secret "error-404-isvc-bbf0f-predictor-serving-cert" not found
Jul 01 17:31:01.339249 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:01.339233    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3121a651-3759-41c0-8363-d5cf6fbb991a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-bbf0f-predictor-85ff6cdb4-jzkmn\" (UID: \"3121a651-3759-41c0-8363-d5cf6fbb991a\") " pod="kserve-ci-e2e-test/error-404-isvc-bbf0f-predictor-85ff6cdb4-jzkmn"
Jul 01 17:31:01.351375 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:01.351314    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb66d\" (UniqueName: \"kubernetes.io/projected/3121a651-3759-41c0-8363-d5cf6fbb991a-kube-api-access-nb66d\") pod \"error-404-isvc-bbf0f-predictor-85ff6cdb4-jzkmn\" (UID: \"3121a651-3759-41c0-8363-d5cf6fbb991a\") " pod="kserve-ci-e2e-test/error-404-isvc-bbf0f-predictor-85ff6cdb4-jzkmn"
Jul 01 17:31:01.497942 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:01.497898    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:31:01.641968 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:01.641873    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/416f5e13-7be6-4ebf-a032-2295aca3c479-proxy-tls\") pod \"success-200-isvc-bbf0f-predictor-6d7cf8c5c-bc4gn\" (UID: \"416f5e13-7be6-4ebf-a032-2295aca3c479\") " pod="kserve-ci-e2e-test/success-200-isvc-bbf0f-predictor-6d7cf8c5c-bc4gn"
Jul 01 17:31:01.644316 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:01.644292    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/416f5e13-7be6-4ebf-a032-2295aca3c479-proxy-tls\") pod \"success-200-isvc-bbf0f-predictor-6d7cf8c5c-bc4gn\" (UID: \"416f5e13-7be6-4ebf-a032-2295aca3c479\") " pod="kserve-ci-e2e-test/success-200-isvc-bbf0f-predictor-6d7cf8c5c-bc4gn"
Jul 01 17:31:01.649693 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:01.649665    2563 generic.go:356] "Generic (PLEG): container finished" podID="73bda8d8-398f-46d6-a526-be667395fad7" containerID="60de77b307e4f9ee24a4428d99b45969256646b7030b7788b13446682d8ed8d5" exitCode=2
Jul 01 17:31:01.649831 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:01.649745    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4517b-predictor-677799f794-kj8t9" event={"ID":"73bda8d8-398f-46d6-a526-be667395fad7","Type":"ContainerDied","Data":"60de77b307e4f9ee24a4428d99b45969256646b7030b7788b13446682d8ed8d5"}
Jul 01 17:31:01.651824 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:01.651801    2563 generic.go:356] "Generic (PLEG): container finished" podID="834629ee-e276-4d11-934b-2b6a7179ee66" containerID="938988ab4bedc0e898ad8ecac1c8f05472a3e6ed1344958b8626959144c3db6e" exitCode=2
Jul 01 17:31:01.651924 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:01.651862    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4517b-predictor-64b9b9f5bc-9pcrm" event={"ID":"834629ee-e276-4d11-934b-2b6a7179ee66","Type":"ContainerDied","Data":"938988ab4bedc0e898ad8ecac1c8f05472a3e6ed1344958b8626959144c3db6e"}
Jul 01 17:31:01.844432 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:01.844382    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3121a651-3759-41c0-8363-d5cf6fbb991a-proxy-tls\") pod \"error-404-isvc-bbf0f-predictor-85ff6cdb4-jzkmn\" (UID: \"3121a651-3759-41c0-8363-d5cf6fbb991a\") " pod="kserve-ci-e2e-test/error-404-isvc-bbf0f-predictor-85ff6cdb4-jzkmn"
Jul 01 17:31:01.847023 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:01.846998    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3121a651-3759-41c0-8363-d5cf6fbb991a-proxy-tls\") pod \"error-404-isvc-bbf0f-predictor-85ff6cdb4-jzkmn\" (UID: \"3121a651-3759-41c0-8363-d5cf6fbb991a\") " pod="kserve-ci-e2e-test/error-404-isvc-bbf0f-predictor-85ff6cdb4-jzkmn"
Jul 01 17:31:01.855958 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:01.855927    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-bbf0f-predictor-6d7cf8c5c-bc4gn"
Jul 01 17:31:01.977123 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:01.977093    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-bbf0f-predictor-85ff6cdb4-jzkmn"
Jul 01 17:31:01.996234 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:01.996095    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bbf0f-predictor-6d7cf8c5c-bc4gn"]
Jul 01 17:31:01.998917 ip-10-0-137-180 kubenswrapper[2563]: W0701 17:31:01.998872    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod416f5e13_7be6_4ebf_a032_2295aca3c479.slice/crio-7a37f6882e923b229c66481b9bfa0d4290cb405fbe027575772f77f1cb7951e0 WatchSource:0}: Error finding container 7a37f6882e923b229c66481b9bfa0d4290cb405fbe027575772f77f1cb7951e0: Status 404 returned error can't find the container with id 7a37f6882e923b229c66481b9bfa0d4290cb405fbe027575772f77f1cb7951e0
Jul 01 17:31:02.129336 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:02.129307    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bbf0f-predictor-85ff6cdb4-jzkmn"]
Jul 01 17:31:02.130902 ip-10-0-137-180 kubenswrapper[2563]: W0701 17:31:02.130874    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3121a651_3759_41c0_8363_d5cf6fbb991a.slice/crio-40aa61ffb28af706f0e05f779a12b596d892d20fbb4318d225e56046cadade30 WatchSource:0}: Error finding container 40aa61ffb28af706f0e05f779a12b596d892d20fbb4318d225e56046cadade30: Status 404 returned error can't find the container with id 40aa61ffb28af706f0e05f779a12b596d892d20fbb4318d225e56046cadade30
Jul 01 17:31:02.221426 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:02.221367    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:31:02.293924 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:02.293874    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4517b-predictor-64b9b9f5bc-9pcrm" podUID="834629ee-e276-4d11-934b-2b6a7179ee66" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Jul 01 17:31:02.323114 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:02.323060    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:31:02.657663 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:02.657627    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bbf0f-predictor-6d7cf8c5c-bc4gn" event={"ID":"416f5e13-7be6-4ebf-a032-2295aca3c479","Type":"ContainerStarted","Data":"739c208d8feb4b2106ce59fc44b9f0de332f128ea886a2e4a0e0b82c9e198fe2"}
Jul 01 17:31:02.657841 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:02.657718    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bbf0f-predictor-6d7cf8c5c-bc4gn" event={"ID":"416f5e13-7be6-4ebf-a032-2295aca3c479","Type":"ContainerStarted","Data":"7efec8161ef8d71d5a9db37da273361ed1e8ca98413cdbe9fc1c29f3962339fa"}
Jul 01 17:31:02.657841 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:02.657748    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bbf0f-predictor-6d7cf8c5c-bc4gn" event={"ID":"416f5e13-7be6-4ebf-a032-2295aca3c479","Type":"ContainerStarted","Data":"7a37f6882e923b229c66481b9bfa0d4290cb405fbe027575772f77f1cb7951e0"}
Jul 01 17:31:02.657841 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:02.657779    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-bbf0f-predictor-6d7cf8c5c-bc4gn"
Jul 01 17:31:02.657841 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:02.657804    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-bbf0f-predictor-6d7cf8c5c-bc4gn"
Jul 01 17:31:02.659279 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:02.659248    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bbf0f-predictor-6d7cf8c5c-bc4gn" podUID="416f5e13-7be6-4ebf-a032-2295aca3c479" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Jul 01 17:31:02.659525 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:02.659503    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bbf0f-predictor-85ff6cdb4-jzkmn" event={"ID":"3121a651-3759-41c0-8363-d5cf6fbb991a","Type":"ContainerStarted","Data":"6d677a44eb29de0b07e88dda900c68c32d3a509e3dc94ffdf0a4cf976e746864"}
Jul 01 17:31:02.659591 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:02.659553    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bbf0f-predictor-85ff6cdb4-jzkmn" event={"ID":"3121a651-3759-41c0-8363-d5cf6fbb991a","Type":"ContainerStarted","Data":"c795b6c5bb040453a5fd6bb908856bf6c40adf5cf6bf710a27dfba3a206b6dad"}
Jul 01 17:31:02.659591 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:02.659585    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-bbf0f-predictor-85ff6cdb4-jzkmn"
Jul 01 17:31:02.659669 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:02.659613    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bbf0f-predictor-85ff6cdb4-jzkmn" event={"ID":"3121a651-3759-41c0-8363-d5cf6fbb991a","Type":"ContainerStarted","Data":"40aa61ffb28af706f0e05f779a12b596d892d20fbb4318d225e56046cadade30"}
Jul 01 17:31:02.659669 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:02.659635    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-bbf0f-predictor-85ff6cdb4-jzkmn"
Jul 01 17:31:02.660603 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:02.660580    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bbf0f-predictor-85ff6cdb4-jzkmn" podUID="3121a651-3759-41c0-8363-d5cf6fbb991a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 01 17:31:02.679196 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:02.679151    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-bbf0f-predictor-6d7cf8c5c-bc4gn" podStartSLOduration=2.679137973 podStartE2EDuration="2.679137973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:31:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:31:02.677684329 +0000 UTC m=+1980.474822741" watchObservedRunningTime="2026-07-01 17:31:02.679137973 +0000 UTC m=+1980.476276374"
Jul 01 17:31:02.697031 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:02.696984    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-bbf0f-predictor-85ff6cdb4-jzkmn" podStartSLOduration=1.696968611 podStartE2EDuration="1.696968611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:31:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:31:02.695616988 +0000 UTC m=+1980.492755388" watchObservedRunningTime="2026-07-01 17:31:02.696968611 +0000 UTC m=+1980.494107011"
Jul 01 17:31:03.663733 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:03.663695    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bbf0f-predictor-85ff6cdb4-jzkmn" podUID="3121a651-3759-41c0-8363-d5cf6fbb991a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 01 17:31:03.664160 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:03.663807    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bbf0f-predictor-6d7cf8c5c-bc4gn" podUID="416f5e13-7be6-4ebf-a032-2295aca3c479" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Jul 01 17:31:04.303661 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:04.303468    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4517b-predictor-677799f794-kj8t9" podUID="73bda8d8-398f-46d6-a526-be667395fad7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Jul 01 17:31:04.388045 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:04.388020    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4517b-predictor-64b9b9f5bc-9pcrm"
Jul 01 17:31:04.433315 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:04.433283    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-db836-predictor-6996c59df8-gc6q8"
Jul 01 17:31:04.465061 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:04.464974    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/834629ee-e276-4d11-934b-2b6a7179ee66-kube-rbac-proxy-sar-config\") pod \"834629ee-e276-4d11-934b-2b6a7179ee66\" (UID: \"834629ee-e276-4d11-934b-2b6a7179ee66\") "
Jul 01 17:31:04.465061 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:04.465011    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-44x89\" (UniqueName: \"kubernetes.io/projected/834629ee-e276-4d11-934b-2b6a7179ee66-kube-api-access-44x89\") pod \"834629ee-e276-4d11-934b-2b6a7179ee66\" (UID: \"834629ee-e276-4d11-934b-2b6a7179ee66\") "
Jul 01 17:31:04.465434 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:04.465401    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/834629ee-e276-4d11-934b-2b6a7179ee66-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "834629ee-e276-4d11-934b-2b6a7179ee66" (UID: "834629ee-e276-4d11-934b-2b6a7179ee66"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:31:04.467256 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:04.467237    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/834629ee-e276-4d11-934b-2b6a7179ee66-kube-api-access-44x89" (OuterVolumeSpecName: "kube-api-access-44x89") pod "834629ee-e276-4d11-934b-2b6a7179ee66" (UID: "834629ee-e276-4d11-934b-2b6a7179ee66"). InnerVolumeSpecName "kube-api-access-44x89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 17:31:04.566371 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:04.566318    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/834629ee-e276-4d11-934b-2b6a7179ee66-proxy-tls\") pod \"834629ee-e276-4d11-934b-2b6a7179ee66\" (UID: \"834629ee-e276-4d11-934b-2b6a7179ee66\") "
Jul 01 17:31:04.566615 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:04.566598    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/834629ee-e276-4d11-934b-2b6a7179ee66-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:31:04.566666 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:04.566618    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44x89\" (UniqueName: \"kubernetes.io/projected/834629ee-e276-4d11-934b-2b6a7179ee66-kube-api-access-44x89\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:31:04.568437 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:04.568405    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/834629ee-e276-4d11-934b-2b6a7179ee66-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "834629ee-e276-4d11-934b-2b6a7179ee66" (UID: "834629ee-e276-4d11-934b-2b6a7179ee66"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 17:31:04.667868 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:04.667839    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/834629ee-e276-4d11-934b-2b6a7179ee66-proxy-tls\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:31:04.670280 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:04.670256    2563 generic.go:356] "Generic (PLEG): container finished" podID="73bda8d8-398f-46d6-a526-be667395fad7" containerID="b88c8d76f8e3a0674ad8135a026550dd7b1e5ba8ccdd40202d2db514407fd16e" exitCode=0
Jul 01 17:31:04.670443 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:04.670324    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4517b-predictor-677799f794-kj8t9" event={"ID":"73bda8d8-398f-46d6-a526-be667395fad7","Type":"ContainerDied","Data":"b88c8d76f8e3a0674ad8135a026550dd7b1e5ba8ccdd40202d2db514407fd16e"}
Jul 01 17:31:04.672018 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:04.671997    2563 generic.go:356] "Generic (PLEG): container finished" podID="834629ee-e276-4d11-934b-2b6a7179ee66" containerID="2a22ccc9d7a74de3ed9f6307c617fb5ad4c9a58d9d5173bf45e3ca8635b03002" exitCode=0
Jul 01 17:31:04.672125 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:04.672050    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4517b-predictor-64b9b9f5bc-9pcrm" event={"ID":"834629ee-e276-4d11-934b-2b6a7179ee66","Type":"ContainerDied","Data":"2a22ccc9d7a74de3ed9f6307c617fb5ad4c9a58d9d5173bf45e3ca8635b03002"}
Jul 01 17:31:04.672125 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:04.672052    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4517b-predictor-64b9b9f5bc-9pcrm"
Jul 01 17:31:04.672125 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:04.672086    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4517b-predictor-64b9b9f5bc-9pcrm" event={"ID":"834629ee-e276-4d11-934b-2b6a7179ee66","Type":"ContainerDied","Data":"d5145e3076658ebe7a51cc878e5fdead81770db4956b2ed5a8615d62e8130a5d"}
Jul 01 17:31:04.672125 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:04.672119    2563 scope.go:117] "RemoveContainer" containerID="938988ab4bedc0e898ad8ecac1c8f05472a3e6ed1344958b8626959144c3db6e"
Jul 01 17:31:04.677072 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:04.677053    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4517b-predictor-677799f794-kj8t9"
Jul 01 17:31:04.680667 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:04.680650    2563 scope.go:117] "RemoveContainer" containerID="2a22ccc9d7a74de3ed9f6307c617fb5ad4c9a58d9d5173bf45e3ca8635b03002"
Jul 01 17:31:04.688145 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:04.688126    2563 scope.go:117] "RemoveContainer" containerID="938988ab4bedc0e898ad8ecac1c8f05472a3e6ed1344958b8626959144c3db6e"
Jul 01 17:31:04.688432 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:31:04.688412    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"938988ab4bedc0e898ad8ecac1c8f05472a3e6ed1344958b8626959144c3db6e\": container with ID starting with 938988ab4bedc0e898ad8ecac1c8f05472a3e6ed1344958b8626959144c3db6e not found: ID does not exist" containerID="938988ab4bedc0e898ad8ecac1c8f05472a3e6ed1344958b8626959144c3db6e"
Jul 01 17:31:04.688523 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:04.688439    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"938988ab4bedc0e898ad8ecac1c8f05472a3e6ed1344958b8626959144c3db6e"} err="failed to get container status \"938988ab4bedc0e898ad8ecac1c8f05472a3e6ed1344958b8626959144c3db6e\": rpc error: code = NotFound desc = could not find container \"938988ab4bedc0e898ad8ecac1c8f05472a3e6ed1344958b8626959144c3db6e\": container with ID starting with 938988ab4bedc0e898ad8ecac1c8f05472a3e6ed1344958b8626959144c3db6e not found: ID does not exist"
Jul 01 17:31:04.688523 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:04.688456    2563 scope.go:117] "RemoveContainer" containerID="2a22ccc9d7a74de3ed9f6307c617fb5ad4c9a58d9d5173bf45e3ca8635b03002"
Jul 01 17:31:04.688674 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:31:04.688660    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a22ccc9d7a74de3ed9f6307c617fb5ad4c9a58d9d5173bf45e3ca8635b03002\": container with ID starting with 2a22ccc9d7a74de3ed9f6307c617fb5ad4c9a58d9d5173bf45e3ca8635b03002 not found: ID does not exist" containerID="2a22ccc9d7a74de3ed9f6307c617fb5ad4c9a58d9d5173bf45e3ca8635b03002"
Jul 01 17:31:04.688714 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:04.688679    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a22ccc9d7a74de3ed9f6307c617fb5ad4c9a58d9d5173bf45e3ca8635b03002"} err="failed to get container status \"2a22ccc9d7a74de3ed9f6307c617fb5ad4c9a58d9d5173bf45e3ca8635b03002\": rpc error: code = NotFound desc = could not find container \"2a22ccc9d7a74de3ed9f6307c617fb5ad4c9a58d9d5173bf45e3ca8635b03002\": container with ID starting with 2a22ccc9d7a74de3ed9f6307c617fb5ad4c9a58d9d5173bf45e3ca8635b03002 not found: ID does not exist"
Jul 01 17:31:04.711882 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:04.711843    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4517b-predictor-64b9b9f5bc-9pcrm"]
Jul 01 17:31:04.715975 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:04.715900    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4517b-predictor-64b9b9f5bc-9pcrm"]
Jul 01 17:31:04.768862 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:04.768817    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ktv2r\" (UniqueName: \"kubernetes.io/projected/73bda8d8-398f-46d6-a526-be667395fad7-kube-api-access-ktv2r\") pod \"73bda8d8-398f-46d6-a526-be667395fad7\" (UID: \"73bda8d8-398f-46d6-a526-be667395fad7\") "
Jul 01 17:31:04.768862 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:04.768867    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73bda8d8-398f-46d6-a526-be667395fad7-kube-rbac-proxy-sar-config\") pod \"73bda8d8-398f-46d6-a526-be667395fad7\" (UID: \"73bda8d8-398f-46d6-a526-be667395fad7\") "
Jul 01 17:31:04.769136 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:04.768897    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73bda8d8-398f-46d6-a526-be667395fad7-proxy-tls\") pod \"73bda8d8-398f-46d6-a526-be667395fad7\" (UID: \"73bda8d8-398f-46d6-a526-be667395fad7\") "
Jul 01 17:31:04.769236 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:04.769208    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73bda8d8-398f-46d6-a526-be667395fad7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "73bda8d8-398f-46d6-a526-be667395fad7" (UID: "73bda8d8-398f-46d6-a526-be667395fad7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:31:04.771130 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:04.771103    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73bda8d8-398f-46d6-a526-be667395fad7-kube-api-access-ktv2r" (OuterVolumeSpecName: "kube-api-access-ktv2r") pod "73bda8d8-398f-46d6-a526-be667395fad7" (UID: "73bda8d8-398f-46d6-a526-be667395fad7"). InnerVolumeSpecName "kube-api-access-ktv2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 17:31:04.771214 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:04.771148    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73bda8d8-398f-46d6-a526-be667395fad7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "73bda8d8-398f-46d6-a526-be667395fad7" (UID: "73bda8d8-398f-46d6-a526-be667395fad7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 17:31:04.870084 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:04.870029    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktv2r\" (UniqueName: \"kubernetes.io/projected/73bda8d8-398f-46d6-a526-be667395fad7-kube-api-access-ktv2r\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:31:04.870084 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:04.870076    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73bda8d8-398f-46d6-a526-be667395fad7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:31:04.870084 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:04.870086    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73bda8d8-398f-46d6-a526-be667395fad7-proxy-tls\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:31:05.680805 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:05.680715    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4517b-predictor-677799f794-kj8t9"
Jul 01 17:31:05.680805 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:05.680736    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4517b-predictor-677799f794-kj8t9" event={"ID":"73bda8d8-398f-46d6-a526-be667395fad7","Type":"ContainerDied","Data":"137df8cf9fc1a952d49e05e37e0b4da843a464424c4d9e3e75a20354193fc062"}
Jul 01 17:31:05.681262 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:05.680865    2563 scope.go:117] "RemoveContainer" containerID="60de77b307e4f9ee24a4428d99b45969256646b7030b7788b13446682d8ed8d5"
Jul 01 17:31:05.689751 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:05.689729    2563 scope.go:117] "RemoveContainer" containerID="b88c8d76f8e3a0674ad8135a026550dd7b1e5ba8ccdd40202d2db514407fd16e"
Jul 01 17:31:05.706852 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:05.706813    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4517b-predictor-677799f794-kj8t9"]
Jul 01 17:31:05.708333 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:05.708310    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4517b-predictor-677799f794-kj8t9"]
Jul 01 17:31:06.493997 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:06.493940    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:31:06.572843 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:06.572806    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:31:06.697534 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:06.697500    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73bda8d8-398f-46d6-a526-be667395fad7" path="/var/lib/kubelet/pods/73bda8d8-398f-46d6-a526-be667395fad7/volumes"
Jul 01 17:31:06.697906 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:06.697893    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="834629ee-e276-4d11-934b-2b6a7179ee66" path="/var/lib/kubelet/pods/834629ee-e276-4d11-934b-2b6a7179ee66/volumes"
Jul 01 17:31:08.669483 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:08.669457    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-bbf0f-predictor-85ff6cdb4-jzkmn"
Jul 01 17:31:08.669928 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:08.669823    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-bbf0f-predictor-6d7cf8c5c-bc4gn"
Jul 01 17:31:08.669983 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:08.669952    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bbf0f-predictor-85ff6cdb4-jzkmn" podUID="3121a651-3759-41c0-8363-d5cf6fbb991a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 01 17:31:08.670196 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:08.670177    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bbf0f-predictor-6d7cf8c5c-bc4gn" podUID="416f5e13-7be6-4ebf-a032-2295aca3c479" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Jul 01 17:31:18.670102 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:18.670051    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bbf0f-predictor-85ff6cdb4-jzkmn" podUID="3121a651-3759-41c0-8363-d5cf6fbb991a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 01 17:31:18.670569 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:18.670206    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bbf0f-predictor-6d7cf8c5c-bc4gn" podUID="416f5e13-7be6-4ebf-a032-2295aca3c479" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Jul 01 17:31:28.670289 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:28.670242    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bbf0f-predictor-85ff6cdb4-jzkmn" podUID="3121a651-3759-41c0-8363-d5cf6fbb991a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 01 17:31:28.670700 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:28.670432    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bbf0f-predictor-6d7cf8c5c-bc4gn" podUID="416f5e13-7be6-4ebf-a032-2295aca3c479" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Jul 01 17:31:38.670519 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:38.670484    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bbf0f-predictor-85ff6cdb4-jzkmn" podUID="3121a651-3759-41c0-8363-d5cf6fbb991a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 01 17:31:38.670964 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:38.670482    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bbf0f-predictor-6d7cf8c5c-bc4gn" podUID="416f5e13-7be6-4ebf-a032-2295aca3c479" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Jul 01 17:31:48.671145 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:48.671105    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-bbf0f-predictor-85ff6cdb4-jzkmn"
Jul 01 17:31:48.671642 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:31:48.671212    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-bbf0f-predictor-6d7cf8c5c-bc4gn"
Jul 01 17:32:01.014120 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:32:01.014058    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:32:01.172934 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:32:01.172902    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:33:57.102739 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:33:57.102660    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:34:26.796065 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:34:26.796026    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:37:25.514322 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:37:25.514230    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:39:20.125944 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:39:20.125904    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:39:30.935203 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:39:30.935162    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:40:15.693667 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:40:15.693630    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:40:15.796716 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:40:15.796685    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bbf0f-predictor-6d7cf8c5c-bc4gn"]
Jul 01 17:40:15.796948 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:40:15.796927    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-bbf0f-predictor-6d7cf8c5c-bc4gn" podUID="416f5e13-7be6-4ebf-a032-2295aca3c479" containerName="kserve-container" containerID="cri-o://7efec8161ef8d71d5a9db37da273361ed1e8ca98413cdbe9fc1c29f3962339fa" gracePeriod=30
Jul 01 17:40:15.797034 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:40:15.796960    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-bbf0f-predictor-6d7cf8c5c-bc4gn" podUID="416f5e13-7be6-4ebf-a032-2295aca3c479" containerName="kube-rbac-proxy" containerID="cri-o://739c208d8feb4b2106ce59fc44b9f0de332f128ea886a2e4a0e0b82c9e198fe2" gracePeriod=30
Jul 01 17:40:15.824649 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:40:15.824613    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:40:15.837738 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:40:15.837705    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bbf0f-predictor-85ff6cdb4-jzkmn"]
Jul 01 17:40:15.837972 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:40:15.837950    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-bbf0f-predictor-85ff6cdb4-jzkmn" podUID="3121a651-3759-41c0-8363-d5cf6fbb991a" containerName="kserve-container" containerID="cri-o://c795b6c5bb040453a5fd6bb908856bf6c40adf5cf6bf710a27dfba3a206b6dad" gracePeriod=30
Jul 01 17:40:15.838051 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:40:15.837999    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-bbf0f-predictor-85ff6cdb4-jzkmn" podUID="3121a651-3759-41c0-8363-d5cf6fbb991a" containerName="kube-rbac-proxy" containerID="cri-o://6d677a44eb29de0b07e88dda900c68c32d3a509e3dc94ffdf0a4cf976e746864" gracePeriod=30
Jul 01 17:40:15.892207 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:40:15.892155    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:40:16.854093 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:40:16.854060    2563 generic.go:356] "Generic (PLEG): container finished" podID="416f5e13-7be6-4ebf-a032-2295aca3c479" containerID="739c208d8feb4b2106ce59fc44b9f0de332f128ea886a2e4a0e0b82c9e198fe2" exitCode=2
Jul 01 17:40:16.854563 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:40:16.854135    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bbf0f-predictor-6d7cf8c5c-bc4gn" event={"ID":"416f5e13-7be6-4ebf-a032-2295aca3c479","Type":"ContainerDied","Data":"739c208d8feb4b2106ce59fc44b9f0de332f128ea886a2e4a0e0b82c9e198fe2"}
Jul 01 17:40:16.855973 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:40:16.855950    2563 generic.go:356] "Generic (PLEG): container finished" podID="3121a651-3759-41c0-8363-d5cf6fbb991a" containerID="6d677a44eb29de0b07e88dda900c68c32d3a509e3dc94ffdf0a4cf976e746864" exitCode=2
Jul 01 17:40:16.856080 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:40:16.855986    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bbf0f-predictor-85ff6cdb4-jzkmn" event={"ID":"3121a651-3759-41c0-8363-d5cf6fbb991a","Type":"ContainerDied","Data":"6d677a44eb29de0b07e88dda900c68c32d3a509e3dc94ffdf0a4cf976e746864"}
Jul 01 17:40:18.664775 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:40:18.664734    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bbf0f-predictor-6d7cf8c5c-bc4gn" podUID="416f5e13-7be6-4ebf-a032-2295aca3c479" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.42:8643/healthz\": dial tcp 10.132.0.42:8643: connect: connection refused"
Jul 01 17:40:18.665148 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:40:18.664742    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bbf0f-predictor-85ff6cdb4-jzkmn" podUID="3121a651-3759-41c0-8363-d5cf6fbb991a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.43:8643/healthz\": dial tcp 10.132.0.43:8643: connect: connection refused"
Jul 01 17:40:18.670194 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:40:18.670173    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bbf0f-predictor-6d7cf8c5c-bc4gn" podUID="416f5e13-7be6-4ebf-a032-2295aca3c479" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Jul 01 17:40:18.670247 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:40:18.670175    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bbf0f-predictor-85ff6cdb4-jzkmn" podUID="3121a651-3759-41c0-8363-d5cf6fbb991a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 01 17:40:19.061425 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:40:19.061396    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-bbf0f-predictor-6d7cf8c5c-bc4gn"
Jul 01 17:40:19.103246 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:40:19.103219    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-bbf0f-predictor-85ff6cdb4-jzkmn"
Jul 01 17:40:19.164598 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:40:19.164567    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3121a651-3759-41c0-8363-d5cf6fbb991a-proxy-tls\") pod \"3121a651-3759-41c0-8363-d5cf6fbb991a\" (UID: \"3121a651-3759-41c0-8363-d5cf6fbb991a\") "
Jul 01 17:40:19.164785 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:40:19.164647    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/416f5e13-7be6-4ebf-a032-2295aca3c479-kube-rbac-proxy-sar-config\") pod \"416f5e13-7be6-4ebf-a032-2295aca3c479\" (UID: \"416f5e13-7be6-4ebf-a032-2295aca3c479\") "
Jul 01 17:40:19.164785 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:40:19.164686    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nb66d\" (UniqueName: \"kubernetes.io/projected/3121a651-3759-41c0-8363-d5cf6fbb991a-kube-api-access-nb66d\") pod \"3121a651-3759-41c0-8363-d5cf6fbb991a\" (UID: \"3121a651-3759-41c0-8363-d5cf6fbb991a\") "
Jul 01 17:40:19.164785 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:40:19.164703    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/416f5e13-7be6-4ebf-a032-2295aca3c479-proxy-tls\") pod \"416f5e13-7be6-4ebf-a032-2295aca3c479\" (UID: \"416f5e13-7be6-4ebf-a032-2295aca3c479\") "
Jul 01 17:40:19.164785 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:40:19.164729    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3121a651-3759-41c0-8363-d5cf6fbb991a-kube-rbac-proxy-sar-config\") pod \"3121a651-3759-41c0-8363-d5cf6fbb991a\" (UID: \"3121a651-3759-41c0-8363-d5cf6fbb991a\") "
Jul 01 17:40:19.165043 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:40:19.164846    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gx9bt\" (UniqueName: \"kubernetes.io/projected/416f5e13-7be6-4ebf-a032-2295aca3c479-kube-api-access-gx9bt\") pod \"416f5e13-7be6-4ebf-a032-2295aca3c479\" (UID: \"416f5e13-7be6-4ebf-a032-2295aca3c479\") "
Jul 01 17:40:19.165125 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:40:19.165060    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/416f5e13-7be6-4ebf-a032-2295aca3c479-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "416f5e13-7be6-4ebf-a032-2295aca3c479" (UID: "416f5e13-7be6-4ebf-a032-2295aca3c479"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:40:19.165125 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:40:19.165066    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3121a651-3759-41c0-8363-d5cf6fbb991a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3121a651-3759-41c0-8363-d5cf6fbb991a" (UID: "3121a651-3759-41c0-8363-d5cf6fbb991a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:40:19.165227 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:40:19.165133    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/416f5e13-7be6-4ebf-a032-2295aca3c479-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:40:19.165227 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:40:19.165152    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3121a651-3759-41c0-8363-d5cf6fbb991a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:40:19.166955 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:40:19.166926    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3121a651-3759-41c0-8363-d5cf6fbb991a-kube-api-access-nb66d" (OuterVolumeSpecName: "kube-api-access-nb66d") pod "3121a651-3759-41c0-8363-d5cf6fbb991a" (UID: "3121a651-3759-41c0-8363-d5cf6fbb991a"). InnerVolumeSpecName "kube-api-access-nb66d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 17:40:19.167067 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:40:19.166995    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/416f5e13-7be6-4ebf-a032-2295aca3c479-kube-api-access-gx9bt" (OuterVolumeSpecName: "kube-api-access-gx9bt") pod "416f5e13-7be6-4ebf-a032-2295aca3c479" (UID: "416f5e13-7be6-4ebf-a032-2295aca3c479"). InnerVolumeSpecName "kube-api-access-gx9bt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 17:40:19.167299 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:40:19.167262    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3121a651-3759-41c0-8363-d5cf6fbb991a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3121a651-3759-41c0-8363-d5cf6fbb991a" (UID: "3121a651-3759-41c0-8363-d5cf6fbb991a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 17:40:19.167461 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:40:19.167262    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/416f5e13-7be6-4ebf-a032-2295aca3c479-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "416f5e13-7be6-4ebf-a032-2295aca3c479" (UID: "416f5e13-7be6-4ebf-a032-2295aca3c479"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 17:40:19.265844 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:40:19.265809    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3121a651-3759-41c0-8363-d5cf6fbb991a-proxy-tls\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:40:19.265844 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:40:19.265842    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nb66d\" (UniqueName: \"kubernetes.io/projected/3121a651-3759-41c0-8363-d5cf6fbb991a-kube-api-access-nb66d\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:40:19.265844 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:40:19.265853    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/416f5e13-7be6-4ebf-a032-2295aca3c479-proxy-tls\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:40:19.266082 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:40:19.265862    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gx9bt\" (UniqueName: \"kubernetes.io/projected/416f5e13-7be6-4ebf-a032-2295aca3c479-kube-api-access-gx9bt\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:40:19.868858 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:40:19.868818    2563 generic.go:356] "Generic (PLEG): container finished" podID="416f5e13-7be6-4ebf-a032-2295aca3c479" containerID="7efec8161ef8d71d5a9db37da273361ed1e8ca98413cdbe9fc1c29f3962339fa" exitCode=0
Jul 01 17:40:19.869270 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:40:19.868876    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-bbf0f-predictor-6d7cf8c5c-bc4gn"
Jul 01 17:40:19.869270 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:40:19.868894    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bbf0f-predictor-6d7cf8c5c-bc4gn" event={"ID":"416f5e13-7be6-4ebf-a032-2295aca3c479","Type":"ContainerDied","Data":"7efec8161ef8d71d5a9db37da273361ed1e8ca98413cdbe9fc1c29f3962339fa"}
Jul 01 17:40:19.869270 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:40:19.868954    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bbf0f-predictor-6d7cf8c5c-bc4gn" event={"ID":"416f5e13-7be6-4ebf-a032-2295aca3c479","Type":"ContainerDied","Data":"7a37f6882e923b229c66481b9bfa0d4290cb405fbe027575772f77f1cb7951e0"}
Jul 01 17:40:19.869270 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:40:19.868984    2563 scope.go:117] "RemoveContainer" containerID="739c208d8feb4b2106ce59fc44b9f0de332f128ea886a2e4a0e0b82c9e198fe2"
Jul 01 17:40:19.870548 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:40:19.870522    2563 generic.go:356] "Generic (PLEG): container finished" podID="3121a651-3759-41c0-8363-d5cf6fbb991a" containerID="c795b6c5bb040453a5fd6bb908856bf6c40adf5cf6bf710a27dfba3a206b6dad" exitCode=0
Jul 01 17:40:19.870656 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:40:19.870587    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-bbf0f-predictor-85ff6cdb4-jzkmn"
Jul 01 17:40:19.870656 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:40:19.870627    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bbf0f-predictor-85ff6cdb4-jzkmn" event={"ID":"3121a651-3759-41c0-8363-d5cf6fbb991a","Type":"ContainerDied","Data":"c795b6c5bb040453a5fd6bb908856bf6c40adf5cf6bf710a27dfba3a206b6dad"}
Jul 01 17:40:19.870743 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:40:19.870691    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bbf0f-predictor-85ff6cdb4-jzkmn" event={"ID":"3121a651-3759-41c0-8363-d5cf6fbb991a","Type":"ContainerDied","Data":"40aa61ffb28af706f0e05f779a12b596d892d20fbb4318d225e56046cadade30"}
Jul 01 17:40:19.878440 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:40:19.878207    2563 scope.go:117] "RemoveContainer" containerID="7efec8161ef8d71d5a9db37da273361ed1e8ca98413cdbe9fc1c29f3962339fa"
Jul 01 17:40:19.886427 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:40:19.886405    2563 scope.go:117] "RemoveContainer" containerID="739c208d8feb4b2106ce59fc44b9f0de332f128ea886a2e4a0e0b82c9e198fe2"
Jul 01 17:40:19.886739 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:40:19.886716    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"739c208d8feb4b2106ce59fc44b9f0de332f128ea886a2e4a0e0b82c9e198fe2\": container with ID starting with 739c208d8feb4b2106ce59fc44b9f0de332f128ea886a2e4a0e0b82c9e198fe2 not found: ID does not exist" containerID="739c208d8feb4b2106ce59fc44b9f0de332f128ea886a2e4a0e0b82c9e198fe2"
Jul 01 17:40:19.886814 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:40:19.886751    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"739c208d8feb4b2106ce59fc44b9f0de332f128ea886a2e4a0e0b82c9e198fe2"} err="failed to get container status \"739c208d8feb4b2106ce59fc44b9f0de332f128ea886a2e4a0e0b82c9e198fe2\": rpc error: code = NotFound desc = could not find container \"739c208d8feb4b2106ce59fc44b9f0de332f128ea886a2e4a0e0b82c9e198fe2\": container with ID starting with 739c208d8feb4b2106ce59fc44b9f0de332f128ea886a2e4a0e0b82c9e198fe2 not found: ID does not exist"
Jul 01 17:40:19.886814 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:40:19.886775    2563 scope.go:117] "RemoveContainer" containerID="7efec8161ef8d71d5a9db37da273361ed1e8ca98413cdbe9fc1c29f3962339fa"
Jul 01 17:40:19.887059 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:40:19.887042    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7efec8161ef8d71d5a9db37da273361ed1e8ca98413cdbe9fc1c29f3962339fa\": container with ID starting with 7efec8161ef8d71d5a9db37da273361ed1e8ca98413cdbe9fc1c29f3962339fa not found: ID does not exist" containerID="7efec8161ef8d71d5a9db37da273361ed1e8ca98413cdbe9fc1c29f3962339fa"
Jul 01 17:40:19.887097 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:40:19.887065    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7efec8161ef8d71d5a9db37da273361ed1e8ca98413cdbe9fc1c29f3962339fa"} err="failed to get container status \"7efec8161ef8d71d5a9db37da273361ed1e8ca98413cdbe9fc1c29f3962339fa\": rpc error: code = NotFound desc = could not find container \"7efec8161ef8d71d5a9db37da273361ed1e8ca98413cdbe9fc1c29f3962339fa\": container with ID starting with 7efec8161ef8d71d5a9db37da273361ed1e8ca98413cdbe9fc1c29f3962339fa not found: ID does not exist"
Jul 01 17:40:19.887097 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:40:19.887079    2563 scope.go:117] "RemoveContainer" containerID="6d677a44eb29de0b07e88dda900c68c32d3a509e3dc94ffdf0a4cf976e746864"
Jul 01 17:40:19.894820 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:40:19.894803    2563 scope.go:117] "RemoveContainer" containerID="c795b6c5bb040453a5fd6bb908856bf6c40adf5cf6bf710a27dfba3a206b6dad"
Jul 01 17:40:19.902511 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:40:19.902490    2563 scope.go:117] "RemoveContainer" containerID="6d677a44eb29de0b07e88dda900c68c32d3a509e3dc94ffdf0a4cf976e746864"
Jul 01 17:40:19.902795 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:40:19.902774    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d677a44eb29de0b07e88dda900c68c32d3a509e3dc94ffdf0a4cf976e746864\": container with ID starting with 6d677a44eb29de0b07e88dda900c68c32d3a509e3dc94ffdf0a4cf976e746864 not found: ID does not exist" containerID="6d677a44eb29de0b07e88dda900c68c32d3a509e3dc94ffdf0a4cf976e746864"
Jul 01 17:40:19.902883 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:40:19.902803    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bbf0f-predictor-6d7cf8c5c-bc4gn"]
Jul 01 17:40:19.902883 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:40:19.902805    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d677a44eb29de0b07e88dda900c68c32d3a509e3dc94ffdf0a4cf976e746864"} err="failed to get container status \"6d677a44eb29de0b07e88dda900c68c32d3a509e3dc94ffdf0a4cf976e746864\": rpc error: code = NotFound desc = could not find container \"6d677a44eb29de0b07e88dda900c68c32d3a509e3dc94ffdf0a4cf976e746864\": container with ID starting with 6d677a44eb29de0b07e88dda900c68c32d3a509e3dc94ffdf0a4cf976e746864 not found: ID does not exist"
Jul 01 17:40:19.902883 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:40:19.902830    2563 scope.go:117] "RemoveContainer" containerID="c795b6c5bb040453a5fd6bb908856bf6c40adf5cf6bf710a27dfba3a206b6dad"
Jul 01 17:40:19.903089 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:40:19.903073    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c795b6c5bb040453a5fd6bb908856bf6c40adf5cf6bf710a27dfba3a206b6dad\": container with ID starting with c795b6c5bb040453a5fd6bb908856bf6c40adf5cf6bf710a27dfba3a206b6dad not found: ID does not exist" containerID="c795b6c5bb040453a5fd6bb908856bf6c40adf5cf6bf710a27dfba3a206b6dad"
Jul 01 17:40:19.903133 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:40:19.903095    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c795b6c5bb040453a5fd6bb908856bf6c40adf5cf6bf710a27dfba3a206b6dad"} err="failed to get container status \"c795b6c5bb040453a5fd6bb908856bf6c40adf5cf6bf710a27dfba3a206b6dad\": rpc error: code = NotFound desc = could not find container \"c795b6c5bb040453a5fd6bb908856bf6c40adf5cf6bf710a27dfba3a206b6dad\": container with ID starting with c795b6c5bb040453a5fd6bb908856bf6c40adf5cf6bf710a27dfba3a206b6dad not found: ID does not exist"
Jul 01 17:40:19.909478 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:40:19.909451    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bbf0f-predictor-6d7cf8c5c-bc4gn"]
Jul 01 17:40:19.941515 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:40:19.941486    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bbf0f-predictor-85ff6cdb4-jzkmn"]
Jul 01 17:40:19.944529 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:40:19.944505    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bbf0f-predictor-85ff6cdb4-jzkmn"]
Jul 01 17:40:20.696301 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:40:20.696269    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3121a651-3759-41c0-8363-d5cf6fbb991a" path="/var/lib/kubelet/pods/3121a651-3759-41c0-8363-d5cf6fbb991a/volumes"
Jul 01 17:40:20.696724 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:40:20.696710    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="416f5e13-7be6-4ebf-a032-2295aca3c479" path="/var/lib/kubelet/pods/416f5e13-7be6-4ebf-a032-2295aca3c479/volumes"
Jul 01 17:41:32.051188 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:41:32.051154    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:43:50.401250 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:43:50.401211    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:47:25.891774 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:25.891696    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:47:26.037734 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:26.037695    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-db836-predictor-598457c8f9-bc6fv"]
Jul 01 17:47:26.037973 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:26.037935    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-db836-predictor-598457c8f9-bc6fv" podUID="0b838ab0-36ca-443c-a390-0323761a03fc" containerName="kserve-container" containerID="cri-o://2069fe2e4ea3a2187b93817d9f0dfd766087ae84ca465533dc07dcffce19afa3" gracePeriod=30
Jul 01 17:47:26.038046 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:26.037992    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-db836-predictor-598457c8f9-bc6fv" podUID="0b838ab0-36ca-443c-a390-0323761a03fc" containerName="kube-rbac-proxy" containerID="cri-o://9c9d4d7f6001b89e3792303b95f304daa74c6e55b3d2754001c585c5472d66ed" gracePeriod=30
Jul 01 17:47:26.062303 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:26.062274    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:47:26.152365 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:26.152313    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-db836-predictor-6996c59df8-gc6q8"]
Jul 01 17:47:26.152650 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:26.152617    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-db836-predictor-6996c59df8-gc6q8" podUID="9a6ed0aa-189f-4c19-bfee-d052a47aa0e9" containerName="kserve-container" containerID="cri-o://dda22b062e8a5075d2ad285e947298bfd9c2ec6f4d00f77161363b8bc629c2a0" gracePeriod=30
Jul 01 17:47:26.152730 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:26.152675    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-db836-predictor-6996c59df8-gc6q8" podUID="9a6ed0aa-189f-4c19-bfee-d052a47aa0e9" containerName="kube-rbac-proxy" containerID="cri-o://adb088884152a4fda39fb8cde0a146a2d3139e38506b3c87c3acf00c616253fd" gracePeriod=30
Jul 01 17:47:26.170610 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:26.170564    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:47:26.507283 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:26.507250    2563 generic.go:356] "Generic (PLEG): container finished" podID="0b838ab0-36ca-443c-a390-0323761a03fc" containerID="9c9d4d7f6001b89e3792303b95f304daa74c6e55b3d2754001c585c5472d66ed" exitCode=2
Jul 01 17:47:26.507472 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:26.507321    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-db836-predictor-598457c8f9-bc6fv" event={"ID":"0b838ab0-36ca-443c-a390-0323761a03fc","Type":"ContainerDied","Data":"9c9d4d7f6001b89e3792303b95f304daa74c6e55b3d2754001c585c5472d66ed"}
Jul 01 17:47:26.509082 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:26.509058    2563 generic.go:356] "Generic (PLEG): container finished" podID="9a6ed0aa-189f-4c19-bfee-d052a47aa0e9" containerID="adb088884152a4fda39fb8cde0a146a2d3139e38506b3c87c3acf00c616253fd" exitCode=2
Jul 01 17:47:26.509200 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:26.509089    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-db836-predictor-6996c59df8-gc6q8" event={"ID":"9a6ed0aa-189f-4c19-bfee-d052a47aa0e9","Type":"ContainerDied","Data":"adb088884152a4fda39fb8cde0a146a2d3139e38506b3c87c3acf00c616253fd"}
Jul 01 17:47:27.250559 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:27.250521    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:47:27.387418 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:27.387386    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:47:29.184563 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:29.184535    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-db836-predictor-598457c8f9-bc6fv"
Jul 01 17:47:29.343035 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:29.343002    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s79fz\" (UniqueName: \"kubernetes.io/projected/0b838ab0-36ca-443c-a390-0323761a03fc-kube-api-access-s79fz\") pod \"0b838ab0-36ca-443c-a390-0323761a03fc\" (UID: \"0b838ab0-36ca-443c-a390-0323761a03fc\") "
Jul 01 17:47:29.343207 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:29.343066    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b838ab0-36ca-443c-a390-0323761a03fc-kube-rbac-proxy-sar-config\") pod \"0b838ab0-36ca-443c-a390-0323761a03fc\" (UID: \"0b838ab0-36ca-443c-a390-0323761a03fc\") "
Jul 01 17:47:29.343207 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:29.343088    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b838ab0-36ca-443c-a390-0323761a03fc-proxy-tls\") pod \"0b838ab0-36ca-443c-a390-0323761a03fc\" (UID: \"0b838ab0-36ca-443c-a390-0323761a03fc\") "
Jul 01 17:47:29.343452 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:29.343427    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b838ab0-36ca-443c-a390-0323761a03fc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0b838ab0-36ca-443c-a390-0323761a03fc" (UID: "0b838ab0-36ca-443c-a390-0323761a03fc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:47:29.345317 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:29.345291    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b838ab0-36ca-443c-a390-0323761a03fc-kube-api-access-s79fz" (OuterVolumeSpecName: "kube-api-access-s79fz") pod "0b838ab0-36ca-443c-a390-0323761a03fc" (UID: "0b838ab0-36ca-443c-a390-0323761a03fc"). InnerVolumeSpecName "kube-api-access-s79fz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 17:47:29.345410 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:29.345295    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b838ab0-36ca-443c-a390-0323761a03fc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0b838ab0-36ca-443c-a390-0323761a03fc" (UID: "0b838ab0-36ca-443c-a390-0323761a03fc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 17:47:29.444165 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:29.444133    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b838ab0-36ca-443c-a390-0323761a03fc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:47:29.444165 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:29.444163    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b838ab0-36ca-443c-a390-0323761a03fc-proxy-tls\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:47:29.444368 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:29.444172    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s79fz\" (UniqueName: \"kubernetes.io/projected/0b838ab0-36ca-443c-a390-0323761a03fc-kube-api-access-s79fz\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:47:29.490042 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:29.490016    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-db836-predictor-6996c59df8-gc6q8"
Jul 01 17:47:29.523851 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:29.523817    2563 generic.go:356] "Generic (PLEG): container finished" podID="9a6ed0aa-189f-4c19-bfee-d052a47aa0e9" containerID="dda22b062e8a5075d2ad285e947298bfd9c2ec6f4d00f77161363b8bc629c2a0" exitCode=0
Jul 01 17:47:29.524038 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:29.523867    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-db836-predictor-6996c59df8-gc6q8"
Jul 01 17:47:29.524038 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:29.523882    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-db836-predictor-6996c59df8-gc6q8" event={"ID":"9a6ed0aa-189f-4c19-bfee-d052a47aa0e9","Type":"ContainerDied","Data":"dda22b062e8a5075d2ad285e947298bfd9c2ec6f4d00f77161363b8bc629c2a0"}
Jul 01 17:47:29.524038 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:29.523966    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-db836-predictor-6996c59df8-gc6q8" event={"ID":"9a6ed0aa-189f-4c19-bfee-d052a47aa0e9","Type":"ContainerDied","Data":"6eece7c8fb4fbc268821c83a612ae831ff9fc8031500b356ecc07b9fdc1bc2d2"}
Jul 01 17:47:29.524038 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:29.524012    2563 scope.go:117] "RemoveContainer" containerID="adb088884152a4fda39fb8cde0a146a2d3139e38506b3c87c3acf00c616253fd"
Jul 01 17:47:29.525643 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:29.525621    2563 generic.go:356] "Generic (PLEG): container finished" podID="0b838ab0-36ca-443c-a390-0323761a03fc" containerID="2069fe2e4ea3a2187b93817d9f0dfd766087ae84ca465533dc07dcffce19afa3" exitCode=0
Jul 01 17:47:29.525775 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:29.525658    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-db836-predictor-598457c8f9-bc6fv" event={"ID":"0b838ab0-36ca-443c-a390-0323761a03fc","Type":"ContainerDied","Data":"2069fe2e4ea3a2187b93817d9f0dfd766087ae84ca465533dc07dcffce19afa3"}
Jul 01 17:47:29.525775 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:29.525690    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-db836-predictor-598457c8f9-bc6fv"
Jul 01 17:47:29.525775 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:29.525723    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-db836-predictor-598457c8f9-bc6fv" event={"ID":"0b838ab0-36ca-443c-a390-0323761a03fc","Type":"ContainerDied","Data":"051f3f246a86658332d98b36320092693536048c10bad5a40b87334f9cb9e141"}
Jul 01 17:47:29.532498 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:29.532479    2563 scope.go:117] "RemoveContainer" containerID="dda22b062e8a5075d2ad285e947298bfd9c2ec6f4d00f77161363b8bc629c2a0"
Jul 01 17:47:29.540817 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:29.540792    2563 scope.go:117] "RemoveContainer" containerID="adb088884152a4fda39fb8cde0a146a2d3139e38506b3c87c3acf00c616253fd"
Jul 01 17:47:29.541121 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:47:29.541103    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adb088884152a4fda39fb8cde0a146a2d3139e38506b3c87c3acf00c616253fd\": container with ID starting with adb088884152a4fda39fb8cde0a146a2d3139e38506b3c87c3acf00c616253fd not found: ID does not exist" containerID="adb088884152a4fda39fb8cde0a146a2d3139e38506b3c87c3acf00c616253fd"
Jul 01 17:47:29.541190 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:29.541131    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adb088884152a4fda39fb8cde0a146a2d3139e38506b3c87c3acf00c616253fd"} err="failed to get container status \"adb088884152a4fda39fb8cde0a146a2d3139e38506b3c87c3acf00c616253fd\": rpc error: code = NotFound desc = could not find container \"adb088884152a4fda39fb8cde0a146a2d3139e38506b3c87c3acf00c616253fd\": container with ID starting with adb088884152a4fda39fb8cde0a146a2d3139e38506b3c87c3acf00c616253fd not found: ID does not exist"
Jul 01 17:47:29.541190 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:29.541148    2563 scope.go:117] "RemoveContainer" containerID="dda22b062e8a5075d2ad285e947298bfd9c2ec6f4d00f77161363b8bc629c2a0"
Jul 01 17:47:29.541389 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:47:29.541368    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dda22b062e8a5075d2ad285e947298bfd9c2ec6f4d00f77161363b8bc629c2a0\": container with ID starting with dda22b062e8a5075d2ad285e947298bfd9c2ec6f4d00f77161363b8bc629c2a0 not found: ID does not exist" containerID="dda22b062e8a5075d2ad285e947298bfd9c2ec6f4d00f77161363b8bc629c2a0"
Jul 01 17:47:29.541433 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:29.541398    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dda22b062e8a5075d2ad285e947298bfd9c2ec6f4d00f77161363b8bc629c2a0"} err="failed to get container status \"dda22b062e8a5075d2ad285e947298bfd9c2ec6f4d00f77161363b8bc629c2a0\": rpc error: code = NotFound desc = could not find container \"dda22b062e8a5075d2ad285e947298bfd9c2ec6f4d00f77161363b8bc629c2a0\": container with ID starting with dda22b062e8a5075d2ad285e947298bfd9c2ec6f4d00f77161363b8bc629c2a0 not found: ID does not exist"
Jul 01 17:47:29.541433 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:29.541415    2563 scope.go:117] "RemoveContainer" containerID="9c9d4d7f6001b89e3792303b95f304daa74c6e55b3d2754001c585c5472d66ed"
Jul 01 17:47:29.549001 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:29.548982    2563 scope.go:117] "RemoveContainer" containerID="2069fe2e4ea3a2187b93817d9f0dfd766087ae84ca465533dc07dcffce19afa3"
Jul 01 17:47:29.551385 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:29.551337    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-db836-predictor-598457c8f9-bc6fv"]
Jul 01 17:47:29.556114 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:29.556090    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-db836-predictor-598457c8f9-bc6fv"]
Jul 01 17:47:29.557304 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:29.557288    2563 scope.go:117] "RemoveContainer" containerID="9c9d4d7f6001b89e3792303b95f304daa74c6e55b3d2754001c585c5472d66ed"
Jul 01 17:47:29.557600 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:47:29.557584    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c9d4d7f6001b89e3792303b95f304daa74c6e55b3d2754001c585c5472d66ed\": container with ID starting with 9c9d4d7f6001b89e3792303b95f304daa74c6e55b3d2754001c585c5472d66ed not found: ID does not exist" containerID="9c9d4d7f6001b89e3792303b95f304daa74c6e55b3d2754001c585c5472d66ed"
Jul 01 17:47:29.557657 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:29.557611    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c9d4d7f6001b89e3792303b95f304daa74c6e55b3d2754001c585c5472d66ed"} err="failed to get container status \"9c9d4d7f6001b89e3792303b95f304daa74c6e55b3d2754001c585c5472d66ed\": rpc error: code = NotFound desc = could not find container \"9c9d4d7f6001b89e3792303b95f304daa74c6e55b3d2754001c585c5472d66ed\": container with ID starting with 9c9d4d7f6001b89e3792303b95f304daa74c6e55b3d2754001c585c5472d66ed not found: ID does not exist"
Jul 01 17:47:29.557657 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:29.557633    2563 scope.go:117] "RemoveContainer" containerID="2069fe2e4ea3a2187b93817d9f0dfd766087ae84ca465533dc07dcffce19afa3"
Jul 01 17:47:29.557866 ip-10-0-137-180 kubenswrapper[2563]: E0701 17:47:29.557848    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2069fe2e4ea3a2187b93817d9f0dfd766087ae84ca465533dc07dcffce19afa3\": container with ID starting with 2069fe2e4ea3a2187b93817d9f0dfd766087ae84ca465533dc07dcffce19afa3 not found: ID does not exist" containerID="2069fe2e4ea3a2187b93817d9f0dfd766087ae84ca465533dc07dcffce19afa3"
Jul 01 17:47:29.557910 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:29.557872    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2069fe2e4ea3a2187b93817d9f0dfd766087ae84ca465533dc07dcffce19afa3"} err="failed to get container status \"2069fe2e4ea3a2187b93817d9f0dfd766087ae84ca465533dc07dcffce19afa3\": rpc error: code = NotFound desc = could not find container \"2069fe2e4ea3a2187b93817d9f0dfd766087ae84ca465533dc07dcffce19afa3\": container with ID starting with 2069fe2e4ea3a2187b93817d9f0dfd766087ae84ca465533dc07dcffce19afa3 not found: ID does not exist"
Jul 01 17:47:29.645850 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:29.645748    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ghvqs\" (UniqueName: \"kubernetes.io/projected/9a6ed0aa-189f-4c19-bfee-d052a47aa0e9-kube-api-access-ghvqs\") pod \"9a6ed0aa-189f-4c19-bfee-d052a47aa0e9\" (UID: \"9a6ed0aa-189f-4c19-bfee-d052a47aa0e9\") "
Jul 01 17:47:29.645850 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:29.645791    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9a6ed0aa-189f-4c19-bfee-d052a47aa0e9-kube-rbac-proxy-sar-config\") pod \"9a6ed0aa-189f-4c19-bfee-d052a47aa0e9\" (UID: \"9a6ed0aa-189f-4c19-bfee-d052a47aa0e9\") "
Jul 01 17:47:29.645850 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:29.645827    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a6ed0aa-189f-4c19-bfee-d052a47aa0e9-proxy-tls\") pod \"9a6ed0aa-189f-4c19-bfee-d052a47aa0e9\" (UID: \"9a6ed0aa-189f-4c19-bfee-d052a47aa0e9\") "
Jul 01 17:47:29.646197 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:29.646173    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a6ed0aa-189f-4c19-bfee-d052a47aa0e9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9a6ed0aa-189f-4c19-bfee-d052a47aa0e9" (UID: "9a6ed0aa-189f-4c19-bfee-d052a47aa0e9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:47:29.648117 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:29.648086    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a6ed0aa-189f-4c19-bfee-d052a47aa0e9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9a6ed0aa-189f-4c19-bfee-d052a47aa0e9" (UID: "9a6ed0aa-189f-4c19-bfee-d052a47aa0e9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 17:47:29.648234 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:29.648125    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a6ed0aa-189f-4c19-bfee-d052a47aa0e9-kube-api-access-ghvqs" (OuterVolumeSpecName: "kube-api-access-ghvqs") pod "9a6ed0aa-189f-4c19-bfee-d052a47aa0e9" (UID: "9a6ed0aa-189f-4c19-bfee-d052a47aa0e9"). InnerVolumeSpecName "kube-api-access-ghvqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 17:47:29.746270 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:29.746233    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a6ed0aa-189f-4c19-bfee-d052a47aa0e9-proxy-tls\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:47:29.746270 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:29.746263    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghvqs\" (UniqueName: \"kubernetes.io/projected/9a6ed0aa-189f-4c19-bfee-d052a47aa0e9-kube-api-access-ghvqs\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:47:29.746270 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:29.746275    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9a6ed0aa-189f-4c19-bfee-d052a47aa0e9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-180.ec2.internal\" DevicePath \"\""
Jul 01 17:47:29.847467 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:29.847415    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-db836-predictor-6996c59df8-gc6q8"]
Jul 01 17:47:29.851379 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:29.851339    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-db836-predictor-6996c59df8-gc6q8"]
Jul 01 17:47:30.427827 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:30.427720    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-db836-predictor-6996c59df8-gc6q8" podUID="9a6ed0aa-189f-4c19-bfee-d052a47aa0e9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.41:8643/healthz\": context deadline exceeded"
Jul 01 17:47:30.696256 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:30.696170    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b838ab0-36ca-443c-a390-0323761a03fc" path="/var/lib/kubelet/pods/0b838ab0-36ca-443c-a390-0323761a03fc/volumes"
Jul 01 17:47:30.696601 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:30.696588    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a6ed0aa-189f-4c19-bfee-d052a47aa0e9" path="/var/lib/kubelet/pods/9a6ed0aa-189f-4c19-bfee-d052a47aa0e9/volumes"
Jul 01 17:47:58.553555 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:58.553520    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:47:58.731031 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:58.730996    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-76kqn/must-gather-hv44f"]
Jul 01 17:47:58.731303 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:58.731290    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b838ab0-36ca-443c-a390-0323761a03fc" containerName="kserve-container"
Jul 01 17:47:58.731362 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:58.731305    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b838ab0-36ca-443c-a390-0323761a03fc" containerName="kserve-container"
Jul 01 17:47:58.731362 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:58.731317    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3121a651-3759-41c0-8363-d5cf6fbb991a" containerName="kube-rbac-proxy"
Jul 01 17:47:58.731362 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:58.731322    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3121a651-3759-41c0-8363-d5cf6fbb991a" containerName="kube-rbac-proxy"
Jul 01 17:47:58.731362 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:58.731329    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a6ed0aa-189f-4c19-bfee-d052a47aa0e9" containerName="kube-rbac-proxy"
Jul 01 17:47:58.731362 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:58.731335    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a6ed0aa-189f-4c19-bfee-d052a47aa0e9" containerName="kube-rbac-proxy"
Jul 01 17:47:58.731362 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:58.731356    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73bda8d8-398f-46d6-a526-be667395fad7" containerName="kube-rbac-proxy"
Jul 01 17:47:58.731362 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:58.731362    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="73bda8d8-398f-46d6-a526-be667395fad7" containerName="kube-rbac-proxy"
Jul 01 17:47:58.731580 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:58.731372    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="416f5e13-7be6-4ebf-a032-2295aca3c479" containerName="kserve-container"
Jul 01 17:47:58.731580 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:58.731377    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="416f5e13-7be6-4ebf-a032-2295aca3c479" containerName="kserve-container"
Jul 01 17:47:58.731580 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:58.731383    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73bda8d8-398f-46d6-a526-be667395fad7" containerName="kserve-container"
Jul 01 17:47:58.731580 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:58.731388    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="73bda8d8-398f-46d6-a526-be667395fad7" containerName="kserve-container"
Jul 01 17:47:58.731580 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:58.731399    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b838ab0-36ca-443c-a390-0323761a03fc" containerName="kube-rbac-proxy"
Jul 01 17:47:58.731580 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:58.731404    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b838ab0-36ca-443c-a390-0323761a03fc" containerName="kube-rbac-proxy"
Jul 01 17:47:58.731580 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:58.731411    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="834629ee-e276-4d11-934b-2b6a7179ee66" containerName="kube-rbac-proxy"
Jul 01 17:47:58.731580 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:58.731416    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="834629ee-e276-4d11-934b-2b6a7179ee66" containerName="kube-rbac-proxy"
Jul 01 17:47:58.731580 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:58.731422    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a6ed0aa-189f-4c19-bfee-d052a47aa0e9" containerName="kserve-container"
Jul 01 17:47:58.731580 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:58.731426    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a6ed0aa-189f-4c19-bfee-d052a47aa0e9" containerName="kserve-container"
Jul 01 17:47:58.731580 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:58.731432    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="834629ee-e276-4d11-934b-2b6a7179ee66" containerName="kserve-container"
Jul 01 17:47:58.731580 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:58.731437    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="834629ee-e276-4d11-934b-2b6a7179ee66" containerName="kserve-container"
Jul 01 17:47:58.731580 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:58.731441    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3121a651-3759-41c0-8363-d5cf6fbb991a" containerName="kserve-container"
Jul 01 17:47:58.731580 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:58.731446    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3121a651-3759-41c0-8363-d5cf6fbb991a" containerName="kserve-container"
Jul 01 17:47:58.731580 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:58.731452    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="416f5e13-7be6-4ebf-a032-2295aca3c479" containerName="kube-rbac-proxy"
Jul 01 17:47:58.731580 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:58.731457    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="416f5e13-7be6-4ebf-a032-2295aca3c479" containerName="kube-rbac-proxy"
Jul 01 17:47:58.731580 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:58.731498    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a6ed0aa-189f-4c19-bfee-d052a47aa0e9" containerName="kserve-container"
Jul 01 17:47:58.731580 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:58.731505    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a6ed0aa-189f-4c19-bfee-d052a47aa0e9" containerName="kube-rbac-proxy"
Jul 01 17:47:58.731580 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:58.731512    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="834629ee-e276-4d11-934b-2b6a7179ee66" containerName="kube-rbac-proxy"
Jul 01 17:47:58.731580 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:58.731517    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="73bda8d8-398f-46d6-a526-be667395fad7" containerName="kserve-container"
Jul 01 17:47:58.731580 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:58.731523    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="416f5e13-7be6-4ebf-a032-2295aca3c479" containerName="kserve-container"
Jul 01 17:47:58.731580 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:58.731528    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3121a651-3759-41c0-8363-d5cf6fbb991a" containerName="kserve-container"
Jul 01 17:47:58.731580 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:58.731533    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="416f5e13-7be6-4ebf-a032-2295aca3c479" containerName="kube-rbac-proxy"
Jul 01 17:47:58.731580 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:58.731539    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="834629ee-e276-4d11-934b-2b6a7179ee66" containerName="kserve-container"
Jul 01 17:47:58.731580 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:58.731544    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b838ab0-36ca-443c-a390-0323761a03fc" containerName="kube-rbac-proxy"
Jul 01 17:47:58.731580 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:58.731550    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3121a651-3759-41c0-8363-d5cf6fbb991a" containerName="kube-rbac-proxy"
Jul 01 17:47:58.731580 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:58.731556    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b838ab0-36ca-443c-a390-0323761a03fc" containerName="kserve-container"
Jul 01 17:47:58.731580 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:58.731562    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="73bda8d8-398f-46d6-a526-be667395fad7" containerName="kube-rbac-proxy"
Jul 01 17:47:58.734106 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:58.734088    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-76kqn/must-gather-hv44f"
Jul 01 17:47:58.737241 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:58.737216    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-76kqn\"/\"openshift-service-ca.crt\""
Jul 01 17:47:58.738017 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:58.737998    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-76kqn\"/\"kube-root-ca.crt\""
Jul 01 17:47:58.738464 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:58.738447    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-76kqn\"/\"default-dockercfg-rgc45\""
Jul 01 17:47:58.754115 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:58.754080    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plz6v\" (UniqueName: \"kubernetes.io/projected/752c9f79-5968-46a1-80c8-a6073ae17f3e-kube-api-access-plz6v\") pod \"must-gather-hv44f\" (UID: \"752c9f79-5968-46a1-80c8-a6073ae17f3e\") " pod="openshift-must-gather-76kqn/must-gather-hv44f"
Jul 01 17:47:58.754115 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:58.754118    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/752c9f79-5968-46a1-80c8-a6073ae17f3e-must-gather-output\") pod \"must-gather-hv44f\" (UID: \"752c9f79-5968-46a1-80c8-a6073ae17f3e\") " pod="openshift-must-gather-76kqn/must-gather-hv44f"
Jul 01 17:47:58.767616 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:58.767581    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-76kqn/must-gather-hv44f"]
Jul 01 17:47:58.789289 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:58.789259    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:47:58.855460 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:58.855338    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plz6v\" (UniqueName: \"kubernetes.io/projected/752c9f79-5968-46a1-80c8-a6073ae17f3e-kube-api-access-plz6v\") pod \"must-gather-hv44f\" (UID: \"752c9f79-5968-46a1-80c8-a6073ae17f3e\") " pod="openshift-must-gather-76kqn/must-gather-hv44f"
Jul 01 17:47:58.855460 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:58.855402    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/752c9f79-5968-46a1-80c8-a6073ae17f3e-must-gather-output\") pod \"must-gather-hv44f\" (UID: \"752c9f79-5968-46a1-80c8-a6073ae17f3e\") " pod="openshift-must-gather-76kqn/must-gather-hv44f"
Jul 01 17:47:58.855811 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:58.855788    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/752c9f79-5968-46a1-80c8-a6073ae17f3e-must-gather-output\") pod \"must-gather-hv44f\" (UID: \"752c9f79-5968-46a1-80c8-a6073ae17f3e\") " pod="openshift-must-gather-76kqn/must-gather-hv44f"
Jul 01 17:47:58.871078 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:58.871044    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plz6v\" (UniqueName: \"kubernetes.io/projected/752c9f79-5968-46a1-80c8-a6073ae17f3e-kube-api-access-plz6v\") pod \"must-gather-hv44f\" (UID: \"752c9f79-5968-46a1-80c8-a6073ae17f3e\") " pod="openshift-must-gather-76kqn/must-gather-hv44f"
Jul 01 17:47:58.925629 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:58.925586    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:47:59.043072 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:59.043035    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-76kqn/must-gather-hv44f"
Jul 01 17:47:59.180726 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:59.180692    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-76kqn/must-gather-hv44f"]
Jul 01 17:47:59.184553 ip-10-0-137-180 kubenswrapper[2563]: W0701 17:47:59.184529    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod752c9f79_5968_46a1_80c8_a6073ae17f3e.slice/crio-b77efbe8acba5071b91742bea13f592dcb0a6ffecf3678f4ebf48032eaa6bd67 WatchSource:0}: Error finding container b77efbe8acba5071b91742bea13f592dcb0a6ffecf3678f4ebf48032eaa6bd67: Status 404 returned error can't find the container with id b77efbe8acba5071b91742bea13f592dcb0a6ffecf3678f4ebf48032eaa6bd67
Jul 01 17:47:59.186567 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:59.186550    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 01 17:47:59.641008 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:47:59.640973    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-76kqn/must-gather-hv44f" event={"ID":"752c9f79-5968-46a1-80c8-a6073ae17f3e","Type":"ContainerStarted","Data":"b77efbe8acba5071b91742bea13f592dcb0a6ffecf3678f4ebf48032eaa6bd67"}
Jul 01 17:48:00.646760 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:48:00.646628    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-76kqn/must-gather-hv44f" event={"ID":"752c9f79-5968-46a1-80c8-a6073ae17f3e","Type":"ContainerStarted","Data":"e319ee36d411c562ecdfbe6a5d512c2cedfc2ed137a4e462357a1dbaaf8e53e1"}
Jul 01 17:48:00.646760 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:48:00.646703    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-76kqn/must-gather-hv44f" event={"ID":"752c9f79-5968-46a1-80c8-a6073ae17f3e","Type":"ContainerStarted","Data":"7616634c3b5763b450d872028a5c4e49e077d77ef42df9d66dc7af0507ae0e1d"}
Jul 01 17:48:00.667741 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:48:00.667685    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-76kqn/must-gather-hv44f" podStartSLOduration=1.469920136 podStartE2EDuration="2.66766781s" totalImagesPullingTime="1.197747674s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:47:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:48:00.665316815 +0000 UTC m=+2998.462455216" watchObservedRunningTime="2026-07-01 17:48:00.66766781 +0000 UTC m=+2998.464806211"
Jul 01 17:48:01.873902 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:48:01.873874    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-qp9lx_ab1a046b-4d82-418b-9c94-3700fa11dd87/global-pull-secret-syncer/0.log"
Jul 01 17:48:02.081471 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:48:02.081435    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log"
Jul 01 17:48:02.332261 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:48:02.332226    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-180.ec2.internal_4125034d4754060c768adbf37d115970/haproxy/0.log"
Jul 01 17:48:04.607926 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:48:04.607873    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:48:04.708508 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:48:04.708477    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:48:05.287259 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:48:05.287222    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fg79l_7ca1dd82-196f-42ef-ac72-d0a1d9f7551f/konnectivity-agent/0.log" limit=1024
Jul 01 17:48:06.665210 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:48:06.665181    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_eb0a004c-c45c-4ea1-b6e0-6d26433107f0/alertmanager/0.log"
Jul 01 17:48:06.805776 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:48:06.805741    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_eb0a004c-c45c-4ea1-b6e0-6d26433107f0/config-reloader/0.log"
Jul 01 17:48:06.851452 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:48:06.851425    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_eb0a004c-c45c-4ea1-b6e0-6d26433107f0/kube-rbac-proxy-web/0.log"
Jul 01 17:48:06.881360 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:48:06.881322    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_eb0a004c-c45c-4ea1-b6e0-6d26433107f0/kube-rbac-proxy/0.log"
Jul 01 17:48:06.906830 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:48:06.906790    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_eb0a004c-c45c-4ea1-b6e0-6d26433107f0/kube-rbac-proxy-metric/0.log"
Jul 01 17:48:06.931921 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:48:06.931833    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_eb0a004c-c45c-4ea1-b6e0-6d26433107f0/prom-label-proxy/0.log"
Jul 01 17:48:06.959616 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:48:06.959583    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_eb0a004c-c45c-4ea1-b6e0-6d26433107f0/init-config-reloader/0.log"
Jul 01 17:48:07.135228 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:48:07.135168    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7b98cc6cf5-r25z2_e4977c17-0c23-43bd-9f44-89c83eb44868/metrics-server/0.log"
Jul 01 17:48:07.167874 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:48:07.167848    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-cd8656b8-k9ctl_77e56a5e-6149-4f54-bb78-99a00606d99f/monitoring-plugin/0.log"
Jul 01 17:48:07.208699 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:48:07.208611    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-74b92_03a13c17-46c7-4f2d-9c1e-0282a2b5b379/node-exporter/0.log"
Jul 01 17:48:07.233409 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:48:07.233384    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-74b92_03a13c17-46c7-4f2d-9c1e-0282a2b5b379/kube-rbac-proxy/0.log"
Jul 01 17:48:07.264065 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:48:07.264035    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-74b92_03a13c17-46c7-4f2d-9c1e-0282a2b5b379/init-textfile/0.log"
Jul 01 17:48:09.707834 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:48:09.707747    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-76kqn/perf-node-gather-daemonset-hvzgf"]
Jul 01 17:48:09.713634 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:48:09.713605    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-76kqn/perf-node-gather-daemonset-hvzgf"
Jul 01 17:48:09.725653 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:48:09.725622    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-76kqn/perf-node-gather-daemonset-hvzgf"]
Jul 01 17:48:09.755360 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:48:09.755314    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/034dd7a9-6e39-4b56-a368-7206f19d95e6-proc\") pod \"perf-node-gather-daemonset-hvzgf\" (UID: \"034dd7a9-6e39-4b56-a368-7206f19d95e6\") " pod="openshift-must-gather-76kqn/perf-node-gather-daemonset-hvzgf"
Jul 01 17:48:09.755526 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:48:09.755376    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/034dd7a9-6e39-4b56-a368-7206f19d95e6-podres\") pod \"perf-node-gather-daemonset-hvzgf\" (UID: \"034dd7a9-6e39-4b56-a368-7206f19d95e6\") " pod="openshift-must-gather-76kqn/perf-node-gather-daemonset-hvzgf"
Jul 01 17:48:09.755526 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:48:09.755401    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/034dd7a9-6e39-4b56-a368-7206f19d95e6-sys\") pod \"perf-node-gather-daemonset-hvzgf\" (UID: \"034dd7a9-6e39-4b56-a368-7206f19d95e6\") " pod="openshift-must-gather-76kqn/perf-node-gather-daemonset-hvzgf"
Jul 01 17:48:09.755526 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:48:09.755419    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/034dd7a9-6e39-4b56-a368-7206f19d95e6-lib-modules\") pod \"perf-node-gather-daemonset-hvzgf\" (UID: \"034dd7a9-6e39-4b56-a368-7206f19d95e6\") " pod="openshift-must-gather-76kqn/perf-node-gather-daemonset-hvzgf"
Jul 01 17:48:09.755526 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:48:09.755507    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mw92\" (UniqueName: \"kubernetes.io/projected/034dd7a9-6e39-4b56-a368-7206f19d95e6-kube-api-access-8mw92\") pod \"perf-node-gather-daemonset-hvzgf\" (UID: \"034dd7a9-6e39-4b56-a368-7206f19d95e6\") " pod="openshift-must-gather-76kqn/perf-node-gather-daemonset-hvzgf"
Jul 01 17:48:09.856204 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:48:09.856170    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mw92\" (UniqueName: \"kubernetes.io/projected/034dd7a9-6e39-4b56-a368-7206f19d95e6-kube-api-access-8mw92\") pod \"perf-node-gather-daemonset-hvzgf\" (UID: \"034dd7a9-6e39-4b56-a368-7206f19d95e6\") " pod="openshift-must-gather-76kqn/perf-node-gather-daemonset-hvzgf"
Jul 01 17:48:09.856422 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:48:09.856216    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/034dd7a9-6e39-4b56-a368-7206f19d95e6-proc\") pod \"perf-node-gather-daemonset-hvzgf\" (UID: \"034dd7a9-6e39-4b56-a368-7206f19d95e6\") " pod="openshift-must-gather-76kqn/perf-node-gather-daemonset-hvzgf"
Jul 01 17:48:09.856422 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:48:09.856238    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/034dd7a9-6e39-4b56-a368-7206f19d95e6-podres\") pod \"perf-node-gather-daemonset-hvzgf\" (UID: \"034dd7a9-6e39-4b56-a368-7206f19d95e6\") " pod="openshift-must-gather-76kqn/perf-node-gather-daemonset-hvzgf"
Jul 01 17:48:09.856422 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:48:09.856256    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/034dd7a9-6e39-4b56-a368-7206f19d95e6-sys\") pod \"perf-node-gather-daemonset-hvzgf\" (UID: \"034dd7a9-6e39-4b56-a368-7206f19d95e6\") " pod="openshift-must-gather-76kqn/perf-node-gather-daemonset-hvzgf"
Jul 01 17:48:09.856422 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:48:09.856273    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/034dd7a9-6e39-4b56-a368-7206f19d95e6-lib-modules\") pod \"perf-node-gather-daemonset-hvzgf\" (UID: \"034dd7a9-6e39-4b56-a368-7206f19d95e6\") " pod="openshift-must-gather-76kqn/perf-node-gather-daemonset-hvzgf"
Jul 01 17:48:09.856422 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:48:09.856362    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/034dd7a9-6e39-4b56-a368-7206f19d95e6-sys\") pod \"perf-node-gather-daemonset-hvzgf\" (UID: \"034dd7a9-6e39-4b56-a368-7206f19d95e6\") " pod="openshift-must-gather-76kqn/perf-node-gather-daemonset-hvzgf"
Jul 01 17:48:09.856422 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:48:09.856363    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/034dd7a9-6e39-4b56-a368-7206f19d95e6-proc\") pod \"perf-node-gather-daemonset-hvzgf\" (UID: \"034dd7a9-6e39-4b56-a368-7206f19d95e6\") " pod="openshift-must-gather-76kqn/perf-node-gather-daemonset-hvzgf"
Jul 01 17:48:09.856422 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:48:09.856421    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/034dd7a9-6e39-4b56-a368-7206f19d95e6-lib-modules\") pod \"perf-node-gather-daemonset-hvzgf\" (UID: \"034dd7a9-6e39-4b56-a368-7206f19d95e6\") " pod="openshift-must-gather-76kqn/perf-node-gather-daemonset-hvzgf"
Jul 01 17:48:09.856838 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:48:09.856422    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/034dd7a9-6e39-4b56-a368-7206f19d95e6-podres\") pod \"perf-node-gather-daemonset-hvzgf\" (UID: \"034dd7a9-6e39-4b56-a368-7206f19d95e6\") " pod="openshift-must-gather-76kqn/perf-node-gather-daemonset-hvzgf"
Jul 01 17:48:09.867498 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:48:09.867469    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mw92\" (UniqueName: \"kubernetes.io/projected/034dd7a9-6e39-4b56-a368-7206f19d95e6-kube-api-access-8mw92\") pod \"perf-node-gather-daemonset-hvzgf\" (UID: \"034dd7a9-6e39-4b56-a368-7206f19d95e6\") " pod="openshift-must-gather-76kqn/perf-node-gather-daemonset-hvzgf"
Jul 01 17:48:10.026740 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:48:10.026709    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-76kqn/perf-node-gather-daemonset-hvzgf"
Jul 01 17:48:10.205220 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:48:10.205191    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-76kqn/perf-node-gather-daemonset-hvzgf"]
Jul 01 17:48:10.207531 ip-10-0-137-180 kubenswrapper[2563]: W0701 17:48:10.207495    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod034dd7a9_6e39_4b56_a368_7206f19d95e6.slice/crio-09c15ad06faf57d9294cb45c4b07cf7bdd559d94ccb82119919b1522c9fb904f WatchSource:0}: Error finding container 09c15ad06faf57d9294cb45c4b07cf7bdd559d94ccb82119919b1522c9fb904f: Status 404 returned error can't find the container with id 09c15ad06faf57d9294cb45c4b07cf7bdd559d94ccb82119919b1522c9fb904f
Jul 01 17:48:10.703596 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:48:10.703448    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-76kqn/perf-node-gather-daemonset-hvzgf" event={"ID":"034dd7a9-6e39-4b56-a368-7206f19d95e6","Type":"ContainerStarted","Data":"c9b24c80d3ff854fb89cbb3d35dab781498bd73c44ccd69f1ccfa508c5917a65"}
Jul 01 17:48:10.703596 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:48:10.703505    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-76kqn/perf-node-gather-daemonset-hvzgf" event={"ID":"034dd7a9-6e39-4b56-a368-7206f19d95e6","Type":"ContainerStarted","Data":"09c15ad06faf57d9294cb45c4b07cf7bdd559d94ccb82119919b1522c9fb904f"}
Jul 01 17:48:10.703596 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:48:10.703538    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-76kqn/perf-node-gather-daemonset-hvzgf"
Jul 01 17:48:10.733957 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:48:10.733905    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-76kqn/perf-node-gather-daemonset-hvzgf" podStartSLOduration=1.733889659 podStartE2EDuration="1.733889659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 17:48:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:48:10.73273236 +0000 UTC m=+3008.529870782" watchObservedRunningTime="2026-07-01 17:48:10.733889659 +0000 UTC m=+3008.531028060"
Jul 01 17:48:11.144116 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:48:11.144090    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-69d9f679db-hxfv2_e9590c1d-88ae-4902-b0ed-2ce2cf13435b/volume-data-source-validator/0.log"
Jul 01 17:48:12.194799 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:48:12.194757    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wr6x6_6867bb01-cbf2-4ced-8a1e-a6fa15bfa8d7/dns/0.log"
Jul 01 17:48:12.220779 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:48:12.220752    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wr6x6_6867bb01-cbf2-4ced-8a1e-a6fa15bfa8d7/kube-rbac-proxy/0.log"
Jul 01 17:48:12.266098 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:48:12.266075    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5bjn2_242fcfc2-df03-4f81-bd5c-1bf8ccf2d2d4/dns-node-resolver/0.log"
Jul 01 17:48:12.868399 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:48:12.868371    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bjw8j_bf6178b8-7786-4321-9401-ca2ec4315089/node-ca/0.log"
Jul 01 17:48:14.126329 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:48:14.126296    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dm2q7_12b51164-c7a2-464b-8652-077ef495ff1f/serve-healthcheck-canary/0.log"
Jul 01 17:48:14.706155 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:48:14.706126    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jncl2_e0f013d5-f637-41e2-8eb2-b011328fd88b/kube-rbac-proxy/0.log"
Jul 01 17:48:14.750896 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:48:14.750871    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jncl2_e0f013d5-f637-41e2-8eb2-b011328fd88b/exporter/0.log"
Jul 01 17:48:14.799062 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:48:14.799032    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jncl2_e0f013d5-f637-41e2-8eb2-b011328fd88b/extractor/0.log"
Jul 01 17:48:16.716609 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:48:16.716585    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-76kqn/perf-node-gather-daemonset-hvzgf"
Jul 01 17:48:17.406892 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:48:17.406864    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-fd65d7d6-swp72_612710ba-2417-4550-a5a8-d93667f657bf/server/0.log"
Jul 01 17:48:17.594912 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:48:17.594879    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-84b54b7f97-b9kmn_7062d5e0-9b0b-4338-bb97-1a79a522f823/manager/0.log"
Jul 01 17:48:26.139642 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:48:26.139606    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gh58z_0ab87fa2-edd2-4c8b-a666-1efa2c3bae8e/kube-multus-additional-cni-plugins/0.log"
Jul 01 17:48:26.178189 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:48:26.178162    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gh58z_0ab87fa2-edd2-4c8b-a666-1efa2c3bae8e/egress-router-binary-copy/0.log"
Jul 01 17:48:26.228415 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:48:26.228382    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gh58z_0ab87fa2-edd2-4c8b-a666-1efa2c3bae8e/cni-plugins/0.log"
Jul 01 17:48:26.291212 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:48:26.291132    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gh58z_0ab87fa2-edd2-4c8b-a666-1efa2c3bae8e/bond-cni-plugin/0.log"
Jul 01 17:48:26.347543 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:48:26.347511    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gh58z_0ab87fa2-edd2-4c8b-a666-1efa2c3bae8e/routeoverride-cni/0.log"
Jul 01 17:48:26.437577 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:48:26.437549    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gh58z_0ab87fa2-edd2-4c8b-a666-1efa2c3bae8e/whereabouts-cni-bincopy/0.log"
Jul 01 17:48:26.545774 ip-10-0-137-180 kubenswrapper[2563]: I0701 17:48:26.545697    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gh58z_0ab87fa2-edd2-4c8b-a666-1efa2c3bae8e/whereabouts-cni/0.log"